403a2cdde4
Merge pull request #1892 from influxdata/crepererum/move_persistence_windows_code
...
chore: move persistence windows related code into own crate
2021-07-05 13:28:22 +00:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
060689b050
Merge pull request #1872 from influxdata/crepererum/ckpt_in_parquet
...
feat: persist part+db checkpoint in parquets and catalog
2021-07-05 07:49:15 +00:00
d96e15c3f7
docs: explain why we store checkpoints in parquet files
2021-07-05 09:42:46 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
a35b334ee5
Merge pull request #1880 from influxdata/crepererum/db_state_in_grpc
...
feat: expose DB state in gRPC interface
2021-07-05 07:28:10 +00:00
bcf43a3de5
Merge branch 'main' into crepererum/db_state_in_grpc
2021-07-05 07:21:48 +00:00
405a6a691b
feat: intial implementation of #1886 : avoid resort if appropriate
2021-07-02 17:57:48 -04:00
5fe49aa017
feat: add flush guard to PersistenceWindows ( #1883 )
...
* feat: add flush guard to PersistenceWindows
* docs: Update comments based on code review
* fix: fmt
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-07-02 20:15:33 +00:00
b4534883fe
refactor: remove table name from upsert_table ( #1882 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-02 15:22:41 +00:00
fba64a41f5
docs: improve profiling docs ( #1869 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-02 10:47:13 +00:00
54fbb60740
feat: expose DB state in gRPC interface
2021-07-02 11:24:36 +02:00
8386b1528e
Merge pull request #1875 from influxdata/pd-remove-mb-size-limit-checks
...
feat: remove MUB size threshold
2021-07-01 20:08:20 +00:00
404da38d6f
Merge branch 'main' into pd-remove-mb-size-limit-checks
2021-07-01 20:01:32 +00:00
5b00bc69e6
refactor: use Arc<Db> in lifecycle actions ( #1873 )
...
* refactor: use Arc<Db> in lifecycle actions
* chore: review feedback
2021-07-01 19:56:33 +00:00
61917c107f
chore: add test for can_move on row count
2021-07-01 15:49:44 -04:00
91f5478012
feat: remove MUB size threshold
...
Removes the MUB chunk close based on size. Also add a check in lifecycle policy to move if the MUB chunk crosses a default row count threshold.
2021-07-01 14:58:29 -04:00
56c8c8d428
feat: Use separate executor for queries and compactions/moves ( #1870 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:47:50 +00:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
43cabac3ac
feat: don't compact more than row threshold ( #1868 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:31:50 +00:00
07826306ed
fix: Always deduplicate data prior to insertion into the ReadBuffer ( #1863 )
...
* fix: mark ReadBuffer as always deduplicated
* fix: Use compact plans during merge
* docs: Update server/src/db/chunk.rs
Co-authored-by: Nga Tran <ntran@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Nga Tran <ntran@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:23:37 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
68c94283bd
Merge pull request #1841 from influxdata/ntran/dedup_less_concat
...
feat: avoid concat_batches if possible
2021-07-01 16:06:49 +00:00
e03a1a1def
Merge branch 'main' into ntran/dedup_less_concat
2021-07-01 15:59:22 +00:00
26167a9e70
Merge pull request #1867 from influxdata/crepererum/rework_db_init_state_machine
...
refactor: rework DB init state machine
2021-07-01 15:31:10 +00:00
84f2391edd
Merge branch 'main' into crepererum/rework_db_init_state_machine
2021-07-01 15:24:12 +00:00
8fc07cf4f0
fix: correctly evaluate exprs matching disjoint rows
2021-07-01 16:05:09 +01:00
d0afc7a176
refactor: clean up and add a missing else case
2021-07-01 11:00:30 -04:00
5cf623201d
fix: deduplicate the last batch before sending it downstream
2021-07-01 10:45:23 -04:00
7235c7b965
refactor: Remove vestigial execution counters ( #1865 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 14:08:06 +00:00
e1e3163752
refactor: rework DB init state machine
...
Since adding new features like "sequencer replay" or init retries would
make the current code too complex, a refactor is required:
Config:
The config struct now holds a `DatabaseState` which is a simple linear
state machine representing the different stages of the database init.
Init:
The init module now has a fixpoint-loop which looks at the state,
decides what to do based on it and repeats until either the DB is
initialized or an error occured. This also makes it easier to continue
the init process "in the middle", e.g. when the preserved catalog is
broken or the sequencer (e.g. Kafka) could not be reached.
2021-07-01 13:47:51 +02:00
8174af9137
Merge pull request #1856 from influxdata/crepererum/parquet_metadata_protobuf
...
refactor: use protobuf for in-parquet metadata
2021-07-01 08:00:27 +00:00
b817ea88dd
Merge branch 'main' into crepererum/parquet_metadata_protobuf
2021-07-01 07:52:39 +00:00
cc038010cd
feat: add persist_age_threshold to LifecycleRules ( #1853 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 21:27:06 +00:00
cfa06e1497
chore: Add query tests for compacted chunks ( #1861 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 20:59:29 +00:00
e8ef8e2790
chore: Merge branch 'main' into ntran/dedup_less_concat
2021-06-30 16:45:01 -04:00
99093b18fb
Merge pull request #1862 from influxdata/ntran/timeout
...
fix: change timeout to have all tests passed on slow laptop
2021-06-30 20:17:39 +00:00
0d24584ed3
Merge branch 'main' into ntran/timeout
2021-06-30 20:10:18 +00:00
f6731c60d7
fix: change timeout to have all tests passed on slow laptop
2021-06-30 16:04:02 -04:00
ba919726b6
test: unit tests
2021-06-30 15:01:31 -04:00
99a15cd452
refactor: single lifecycle error enumeration ( #1859 )
...
* refactor: single lifecycle error enumeration
* fix: fmt
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-06-30 18:35:57 +00:00
817a480cde
refactor: move lifecycle implementations out of db.rs and into their own modules ( #1858 )
...
* refactor: move lifecycle implementations out of db.rs and into their own modules
* fix: clippy
2021-06-30 17:24:04 +00:00
9e1723620c
refactor: rename load_chunk_to_read_buffer to move_chunk_to_read_buffer ( #1857 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 16:53:18 +00:00
a849c50749
Merge pull request #1848 from influxdata/er/refactor/gRPC_logging
...
refactor: improve logging level for gRPC errors
2021-06-30 15:49:46 +00:00
0beb5a888c
Merge branch 'main' into er/refactor/gRPC_logging
2021-06-30 15:42:03 +00:00
1049cb0a2a
refactor: improve logging level for gRPC errors
2021-06-30 16:41:22 +01:00
2a06b93b00
chore: Merge branch 'main' into ntran/dedup_less_concat
2021-06-30 11:37:15 -04:00
1dbdabd66e
fix: 2 values are also considered to be the same if at least one of them is invalid
2021-06-30 10:52:21 -04:00
4204127b05
refactor: use protobuf for in-parquet metadata
2021-06-30 16:51:37 +02:00
9448b8044e
Merge pull request #1850 from influxdata/crepererum/safer_min_max_sequence
...
refactor: make `MinMaxSequence` safer to use
2021-06-30 14:45:41 +00:00
kodiakhq[bot]
Marco Neumann
Nga Tran
Raphael Taylor-Davies
Marko Mikulicic
Paul Dix
Andrew Lamb
Jacob Marble
Edd Robinson