bde2b2b5df
refactor: `Tick` -> `MakeWritesPersistable`
2021-08-05 14:21:36 +02:00
548145a70e
docs: state that `background_worker_now_override` is for testing only
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-08-05 14:08:24 +02:00
015d858f88
test: add failing regression test for #2185
...
We need a partition that is partially persisted for this.
This requires some rework for the time handling in `Db` to make it
mockable.
The remaining bits are test framework extensions.
2021-08-05 11:44:44 +02:00
dd9beab166
feat: error database if no rules ( #2187 )
2021-08-04 11:58:59 +00:00
60aee3e70c
refactor: avoid copying a sequence
2021-08-04 13:23:30 +02:00
1b2e331ec1
test: extend replay tests a bit
2021-08-04 13:23:30 +02:00
af1edcdcbb
fix: docstrings
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-08-04 13:23:30 +02:00
39f30fd0b6
test: make test queries easier to understand
2021-08-04 13:23:30 +02:00
567ef7e991
test: expland replay tests a bit
2021-08-04 13:23:30 +02:00
b868cd160e
docs: fix code comment about sequence ranges
2021-08-04 13:23:30 +02:00
ed70b73fd8
test: determistic concurreny for `TestDb`
2021-08-04 13:23:30 +02:00
a2bc97b923
feat: prune sequence numbers during replay
...
This only prunes entire sequence numbers, it does not (yet!) prune
individual rows for sequence numbers that are partially persisted.
2021-08-04 13:23:30 +02:00
7a18087044
feat: Log messages during database initialization ( #2180 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-08-04 11:04:41 +00:00
65991270e4
refactor: rename handle and shutdown to link them to background worker
2021-08-04 12:04:47 +02:00
c2faf0876b
fix: fix typo and explain policy storage
2021-08-04 11:55:31 +02:00
42953b0561
fix: increase max wait time for compaction to 60s
2021-08-04 11:51:07 +02:00
164c6e3743
feat: improve hard buffer logging and use that as test assertions
2021-08-04 11:49:05 +02:00
657f469317
test: fix `seek_to_end_works`
2021-08-04 11:33:47 +02:00
6ce1984d75
test: improve hard buffer limit tests
2021-08-04 11:33:47 +02:00
3ac88ffc49
fix: hard buffer limits around write buffer consumption
...
- when reading entries from write buffer during normal playback, do not
throw away entries when hitting the hard buffer limit. instead wait
for compaction to sort it out
- during playback, wait for compaction
2021-08-04 11:33:47 +02:00
9ea04a42ff
refactor: start background worker before performing replay
...
This enables compaction during replay.
2021-08-04 11:33:47 +02:00
0fe8eda89e
refactor: move lifecycle policy into Db struct
2021-08-04 11:33:47 +02:00
98d4c9fca1
feat: switch protobuf write service to canonical definition ( #2182 )
...
* feat: switch protobuf write service to canonical definition
The protobuf definition used for the proto write endpoint was a WIP. Now
that a canonical definition exists at
https://github.com/influxdata/influxdb-pb-data-protocol/ we can switch
to that.
* chore: lint etc
* chore: fix rustdoc nit in proto definition comment
2021-08-04 00:16:49 +00:00
ffb36cd50c
refactor: extract ApplicationState from Server ( #2167 )
...
* refactor: extract Application from Server
* chore: review feedback
2021-08-03 09:36:55 +00:00
f504d6002a
docs: error handling for `seek_to_end`
2021-08-03 09:40:40 +02:00
c912e91c95
feat: add flag to skip replay
...
Closes #2169 .
2021-08-02 18:14:19 +02:00
9d15798288
fix: Address or allow Clippy warnings new with Rust 1.54
2021-07-30 09:59:59 -04:00
545222303f
Merge branch 'main' into cn/cc-only
2021-07-29 17:18:16 +00:00
79a04f861f
refactor: Take chunk and write time when creating a new MUB chunk
...
This makes it more consistent with the API of creating a new read buffer
chunk and a new object store chunk.
2021-07-29 10:11:50 -04:00
431774c8b7
refactor: extract resolver from server::Config ( #2143 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-29 13:14:58 +00:00
336ff30484
refactor: make server fields private ( #2144 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-29 13:06:05 +00:00
df3b162475
refactor: move connection manager to separate module ( #2142 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-29 12:58:15 +00:00
ad0a9549de
fix: Avoid an unnecessary parsing of iox metadata
...
In one case where ParquetChunk::new was being called, the calling code
had just parsed the IoxMetadata too. In the other case, the calling code
had just *created* the IoxMetadata being parsed. In both cases, this
re-parsing wasn't actually needed; the two bits of info
ParquetChunk::new can be easily passed in.
2021-07-28 14:25:56 -04:00
af7866a638
refactor: Remove first/last write times from ParquetFile chunks
2021-07-28 14:12:36 -04:00
9371f781fe
test: add "missing entry" replay test
2021-07-28 17:34:02 +02:00
04e797c706
refactor: pass sequencer numbers directly to DB checkpoint
...
First of all using a partition checkpoint as some kind of intermediate
representation was kinda a hack because partition checkpoints should
only created for to-be-persisted partitions, not for the others.
API-wise it should only be possible to construct a partition checkpoint
from a flush handle.
Also we were only able to construct partition checkpoints for partitions
that had unpersisted data, otherwise there was no sane way to fill the
`min_unpersisted_timestamp`. We must however scan all partitions no
matter if there is unpersisted data so that we can determine the maximum
seen sequence numbers. This was caught by a replay test resulting in a
catalog state where the last database checkpoint had lower maximum seen
sequence numbers than some partition checkpoint, bailing out with an
error.
So overall it turns out that passing the sequencer numbers directly
instead of wrapping them into a partition checkpoint is the better
implementation.
2021-07-28 17:28:34 +02:00
a0764cbafd
test: add failing replay test
2021-07-28 17:28:34 +02:00
29ddc36154
docs: state the reason for some replay tests
2021-07-28 17:28:34 +02:00
ca90e92ecc
fix: replay tests should not fail when awaiting on query results
2021-07-28 17:28:34 +02:00
11b7755325
refactor: Remove first/last write times from RUB chunks
2021-07-28 11:22:22 -04:00
4689b5e4e5
refactor: Remove first/last write times from MUB chunks
2021-07-28 11:02:57 -04:00
0f5398c4b9
refactor: Store first/last write on DbChunk snapshots
2021-07-28 11:02:56 -04:00
bc2ec3338f
refactor: Move MBChunk creation inside CatalogChunk new_open
2021-07-28 11:02:56 -04:00
b5195571fa
refactor: Move MBChunk creation inside partition create_open_chunk
2021-07-28 11:02:56 -04:00
7b73190d79
Merge branch 'main' into crepererum/ingest_wallclock
2021-07-28 13:49:08 +00:00
0fcec6b742
refactor: move ingest timestamp from sequence to sequended entry
2021-07-28 15:40:35 +02:00
754d647c06
feat: enable row timestamp metrics with environment variable ( #2135 )
...
* feat: enable row timestamp metrics with environment variable
* chore: fix test
* chore: fix typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-28 13:39:56 +00:00
8add00e761
feat: Make CatalogChunk first/last write times required
...
Connects to #1927 .
2021-07-28 09:22:06 -04:00
09e48018a0
refactor: Move ts_to_timestamp fn into the only file it's used in
2021-07-28 09:22:06 -04:00
7c9a21632b
refactor: Organize uses
2021-07-28 09:22:04 -04:00
Marco Neumann
Raphael Taylor-Davies
Andrew Lamb
Jacob Marble
Carol (Nichols || Goulding)
kodiakhq[bot]