d928bc84e6
feat: Thread time_of_{first,last}_write through Parquet metadata
2021-07-23 14:07:35 -04:00
3c794153dd
refactor: Organize uses
2021-07-23 13:48:15 -04:00
7de946c534
fix: ChunkStage::WrittenToObjectStore is now called ChunkStage::Persisted
2021-07-23 13:11:42 -04:00
844a025c7c
feat: drop based on LRU ( #2075 ) ( #2092 )
...
* feat: drop based on LRU (#2075 )
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-23 08:31:28 +00:00
53b00ec4e0
test: split replay tests
2021-07-23 10:17:02 +02:00
be1bc7025c
refactor: use a single seek loop during replay
2021-07-23 10:05:11 +02:00
ace247d5c2
feat: add replay logging
2021-07-23 10:03:02 +02:00
0c89930b7c
feat: check that replay plan and write buffer are in-sync
2021-07-23 09:39:46 +02:00
db0f501b02
feat: implement naive replay
2021-07-23 09:24:04 +02:00
6ef3680554
feat: collect replay plan during catalog loading
2021-07-23 09:23:06 +02:00
71f3f1aba2
Merge branch 'main' into cn/refactorings
2021-07-22 19:44:18 +00:00
01c79f1a1a
fix: Print all timestamps using RFC3339 format ( #2098 )
...
* fix: Use IOx pretty printer rather than arrow pretty printer
* chore: update tests in the query crate
* chore: update influxdb_iox tests
* chore: Update end to end tests
* chore: update query_tests
* chore: update mutable_buffer tests
* refactor: update parquet_file tests
* refactor: update db tests
* chore: update kafka integration test output
* fix: merge conflict
2021-07-22 19:04:52 +00:00
20d06e3225
feat: include more information in system.operations table ( #2097 )
...
* feat: include more information in system.operations table
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 17:16:09 +00:00
14cb2a6bef
test: Add assertions for first/last write times as chunks move
2021-07-22 11:35:23 -04:00
37f24ebfc7
feat: Record first/last write times for creation of read_buffer::Chunk
2021-07-22 11:35:23 -04:00
0c44179aa9
feat: Add first/last write time on DbChunk
...
To eventually be used in collect_rub
2021-07-22 11:35:23 -04:00
8d1d877196
feat: Record first/last write times for RUB chunks
2021-07-22 11:35:22 -04:00
28fc01ecee
test: Make test failure messages easier to read
2021-07-22 11:15:19 -04:00
6feea3b2d5
feat: Require at least one RecordBatch to create a read_buffer::Chunk::new
...
In the signature only for the moment.
2021-07-22 11:15:18 -04:00
d347750366
refactor: Make collect_rub create the RBChunk
...
Which gets rid of the need for new_rub_chunk.
This will enable creating RBChunks that are guaranteed to have data.
2021-07-22 11:15:18 -04:00
0a724878e6
refactor: Organize uses
2021-07-22 11:15:18 -04:00
7371b0aabf
refactor: Use existing new_rub_chunk function that has the same code
2021-07-22 11:15:18 -04:00
eadcb3265a
refactor: Use some TryStreamExt adapters in collect_rub
2021-07-22 11:15:18 -04:00
38e375d11a
feat: add chunk storage metrics ( #2069 )
...
* feat: add chunk storage metrics
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 15:13:09 +00:00
8c974beba0
feat: add access timestamps to CatalogChunk ( #2075 ) ( #2081 )
...
* feat: add access timestamps to CatalogChunk (#2075 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 12:19:30 +00:00
8c4f5cb237
Merge branch 'main' into crepererum/fix_db_checkpoints
2021-07-21 16:46:13 +00:00
cddf94653c
refactor: use `write_buffer` subsystem for ingest metrics
2021-07-21 15:07:59 +02:00
fd00206fbb
refactor: increase watermark update frequence to once per 10s
2021-07-21 15:02:48 +02:00
2f1efcf517
docs: clarify difference
2021-07-21 15:00:53 +02:00
4d5f209030
docs: do not repeat unix that often
2021-07-21 14:59:07 +02:00
ec866de193
fix: collect checkpoint data from all tables
2021-07-21 14:48:29 +02:00
7d597d1d5c
refactor: make ingest metrics easier to understand
2021-07-21 13:57:53 +02:00
fb931bb1ca
feat: write buffer ingestion metrics
2021-07-21 11:59:52 +02:00
091837420f
feat: add PersistenceWindows sytem table ( #2030 ) ( #2062 )
...
* feat: add PersistenceWindows sytem table (#2030 )
* chore: update log
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 13:10:57 +00:00
e4d2c51e8b
fix: update PersistenceWindows on rules update ( #2018 ) ( #2060 )
...
* fix: update PersistenceWindows on rules update (#2018 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 12:44:47 +00:00
58dd7e9532
Merge branch 'main' into crepererum/writer_buffer_seek
2021-07-20 12:29:18 +00:00
cf8a60252d
refactor: split system_tables module into smaller modules ( #2061 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 12:19:20 +00:00
ec7ebdff29
refactor: use lifetimes to ensure single stream / no seek while streaming
2021-07-20 13:52:33 +02:00
b0663a0337
feat: disallow multiple write buffer streams and seeking while streams
...
Multiple streams will mess up ordering. Seeking while streaming is
likely a bug and should not work.
2021-07-20 12:35:20 +02:00
767c2a6fe1
refactor: explicit server startup state machine ( #2040 )
...
* refactor: explicit server startup state machine
* chore: update `ServerStage` docs
* chore: further docs
* chore: more logging
* chore: format
2021-07-20 10:11:18 +00:00
5bf68c4a57
Merge branch 'main' into jg/snafu-driveby
2021-07-19 20:20:30 +00:00
1c8c227668
refactor: push database rules update into Db ( #2052 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-19 16:05:21 +00:00
1d1ac12522
Merge branch 'main' into crepererum/write_buffer_multiple_streams
2021-07-19 15:50:42 +00:00
4da8a16c18
chore: update to arrow 5.0 and master datafusion ( #2049 )
...
* chore: update to arrow 5.0 and master datafusion
* fix: Update test for change in object size
2021-07-19 12:49:51 +00:00
e2a23c7ac3
fix: persist deadlock ( #2045 ) ( #2046 )
2021-07-19 11:52:48 +00:00
592424c896
refactor: use one stream per sequencer/partition
...
Advantages are:
- for large DBs w/ many partitions we can ingest data in-parallel
- on top of this change we can implement per-sequencer seeking, which is
required for replay
2021-07-19 12:26:58 +02:00
a1d47a8a7a
Merge branch 'main' into crepererum/simplify_testdb_lifecycle_rules
2021-07-19 09:53:35 +00:00
5fc98c7c56
feat: add failure reporting to TaskTracker ( #2031 )
...
* feat: add failure reporting to TaskTracker
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-19 09:17:20 +00:00
2263189e09
test: make TestDb lifecycle better for testing
...
This is a leftover from #1972 .
2021-07-19 09:50:44 +02:00
449ba46b22
refactor: Make more use of SNAFU's context methods and ensure! macro
2021-07-16 16:31:50 -04:00
Jake Goulding
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Marco Neumann
kodiakhq[bot]
Andrew Lamb