c386ac013c
fix: fix flaky `test_unload_partition_chunk`
...
Do not rely on the fact that the chunk ID is 1, because compaction and
other mechanisms might create chunks using different IDs.
Fixes #2109 .
2021-07-26 12:01:28 +02:00
ceacd6b4e7
test: return chunks from `wait_for[_exact_chunk]_state`
2021-07-26 11:57:36 +02:00
f361e344c3
Merge pull request #2100 from influxdata/sinks
...
refactor: Generailize routing sinks
2021-07-26 09:28:19 +00:00
e5ee252876
feat: Add kafka sink variant
2021-07-26 11:08:02 +02:00
d58a3ccbc7
refactor: Add sink to routing config
...
This deprecates the "target" field in the RoutingConfig and replaces it with the "sink"
field, which has a variant that accepts a node group.
This commit is backward compatible in that it will accept existing configs.
The configs will roundtrip to the new format though (i.e. `database get` will render
the sink field).
2021-07-26 11:08:01 +02:00
16a82ba350
refactor: Generailize sinks: Rename Shard to Sink
...
The ShardConfig applies matchers that resolve to a shard number.
The config then applies a mapping between shard numbers to targets.
The type that encapsulated the target that a shard points to was also called
a "Shard". This is confusing. This commit changes it to "Sink", i.e. a destination
for traffic to go to. Subsequent commits will expand the definition of a Sink to
encompass different kinds of sinks (like kafka write buffer, "devnull", ...)
This changes only the name of the protobuf message and the related rust types,
it doesn't change any name of the json-rendered protobuf configs.
2021-07-26 11:08:00 +02:00
9d408296d8
chore: Update datafusion deps ( #2108 )
...
* chore: Update datafusion deps
* chore: update other deps
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-23 21:24:01 +00:00
c595039c81
feat: add row timestamp metrics ( #2101 )
...
* feat: add row timestamp metrics
* chore: review feedback
2021-07-23 19:17:11 +00:00
446af5eb15
fix: consistent write timestamps ( #2104 )
...
* fix: consistent write timestamps
* chore: fix benchmarks
2021-07-23 18:04:15 +00:00
844a025c7c
feat: drop based on LRU ( #2075 ) ( #2092 )
...
* feat: drop based on LRU (#2075 )
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-23 08:31:28 +00:00
50b436a8a3
Merge pull request #2093 from influxdata/crepererum/naive_replay2
...
feat: implement naive replay
2021-07-23 08:23:57 +00:00
53b00ec4e0
test: split replay tests
2021-07-23 10:17:02 +02:00
be1bc7025c
refactor: use a single seek loop during replay
2021-07-23 10:05:11 +02:00
ace247d5c2
feat: add replay logging
2021-07-23 10:03:02 +02:00
0c89930b7c
feat: check that replay plan and write buffer are in-sync
2021-07-23 09:39:46 +02:00
db0f501b02
feat: implement naive replay
2021-07-23 09:24:04 +02:00
6ef3680554
feat: collect replay plan during catalog loading
2021-07-23 09:23:06 +02:00
cf21c9ec40
feat: impl `Clone` for `ReplayPlan`
2021-07-23 09:23:06 +02:00
38261cc7ac
test: add tests using `to_timestamp()` as predicates in SQL ( #2099 )
...
* test: add tests using `to_timestamp()` as predicates in SQL
* fix: cleanup redundancy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 21:06:52 +00:00
67478066b0
Merge pull request #2074 from influxdata/pd/add-parquet-cache
...
feat: implement skeleton for object store file cache
2021-07-22 20:29:29 +00:00
5b5453a020
Merge branch 'main' into pd/add-parquet-cache
2021-07-22 20:21:53 +00:00
88e29dede9
chore: remove extraneous example code from parquet storage
2021-07-22 16:21:13 -04:00
8f6ca07657
Merge pull request #2029 from influxdata/cn/refactorings
...
feat: Record first/last write times on read buffer chunks
2021-07-22 19:51:14 +00:00
71f3f1aba2
Merge branch 'main' into cn/refactorings
2021-07-22 19:44:18 +00:00
01c79f1a1a
fix: Print all timestamps using RFC3339 format ( #2098 )
...
* fix: Use IOx pretty printer rather than arrow pretty printer
* chore: update tests in the query crate
* chore: update influxdb_iox tests
* chore: Update end to end tests
* chore: update query_tests
* chore: update mutable_buffer tests
* refactor: update parquet_file tests
* refactor: update db tests
* chore: update kafka integration test output
* fix: merge conflict
2021-07-22 19:04:52 +00:00
a27d8fd859
Merge pull request #2087 from influxdata/ntran/fix_test_stats
...
test: fix data and statistics of tests
2021-07-22 18:47:19 +00:00
2fcd3a0b8e
Merge branch 'main' into ntran/fix_test_stats
2021-07-22 18:40:30 +00:00
11ba4b5f6a
fix: fix unit_test setting to have the desired results
2021-07-22 14:22:08 -04:00
20d06e3225
feat: include more information in system.operations table ( #2097 )
...
* feat: include more information in system.operations table
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 17:16:09 +00:00
b3d6e3ed7b
fix: Implement pretty printer wity RFC3339 formatting for timestamps ( #2096 )
...
* fix: Implement pretty printer wity RFC3339 formatting for timestamps
* fix: doc strings + fmt
2021-07-22 17:08:27 +00:00
05782eb980
refactor: Move first/last write times up to read buffer Chunk rather than MetaData
2021-07-22 12:27:46 -04:00
14cb2a6bef
test: Add assertions for first/last write times as chunks move
2021-07-22 11:35:23 -04:00
37f24ebfc7
feat: Record first/last write times for creation of read_buffer::Chunk
2021-07-22 11:35:23 -04:00
4e6b79534b
feat: Require passing first/last write times for creation of Table
2021-07-22 11:35:23 -04:00
b7bedeaaf3
feat: Require passing first/last write times for creation of Table MetaData
2021-07-22 11:35:23 -04:00
0c44179aa9
feat: Add first/last write time on DbChunk
...
To eventually be used in collect_rub
2021-07-22 11:35:23 -04:00
799406be72
refactor: Have MUB chunk snapshots hold a TableSummaryAndTimes rather than only ColumnSummaries
...
This way we can have the first/last write times on MUB chunk snapshots
without needing to deconstruct/reconstruct TableSummaries as it was
doing.
2021-07-22 11:35:23 -04:00
956647d966
docs: Fix an outdated comment
2021-07-22 11:35:23 -04:00
8d1d877196
feat: Record first/last write times for RUB chunks
2021-07-22 11:35:22 -04:00
28fc01ecee
test: Make test failure messages easier to read
2021-07-22 11:15:19 -04:00
16b07e5b31
refactor: Always use Table::with_row_group to ensure Tables are never empty
...
Remove Table::new that created an empty table.
2021-07-22 11:15:18 -04:00
6feea3b2d5
feat: Require at least one RecordBatch to create a read_buffer::Chunk::new
...
In the signature only for the moment.
2021-07-22 11:15:18 -04:00
bbb4462264
refactor: Extract a function for the RecordBatch to RowGroup transformation with logging
...
So that we can call it from RBChunk::new too.
2021-07-22 11:15:18 -04:00
d347750366
refactor: Make collect_rub create the RBChunk
...
Which gets rid of the need for new_rub_chunk.
This will enable creating RBChunks that are guaranteed to have data.
2021-07-22 11:15:18 -04:00
0a724878e6
refactor: Organize uses
2021-07-22 11:15:18 -04:00
7371b0aabf
refactor: Use existing new_rub_chunk function that has the same code
2021-07-22 11:15:18 -04:00
eadcb3265a
refactor: Use some TryStreamExt adapters in collect_rub
2021-07-22 11:15:18 -04:00
38e375d11a
feat: add chunk storage metrics ( #2069 )
...
* feat: add chunk storage metrics
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 15:13:09 +00:00
8c974beba0
feat: add access timestamps to CatalogChunk ( #2075 ) ( #2081 )
...
* feat: add access timestamps to CatalogChunk (#2075 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-22 12:19:30 +00:00
f4b9fe20fd
Merge pull request #2084 from influxdata/crepererum/fix_checkpoints_again
...
refactor: correctly track "seen" ranges in persistence checkpoints
2021-07-22 11:45:39 +00:00
Marco Neumann
kodiakhq[bot]
Marko Mikulicic
Andrew Lamb
Raphael Taylor-Davies
Paul Dix
Nga Tran
Carol (Nichols || Goulding)