3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
bc958e2ff0
refactor: use Arcs to pass schemas around
2021-07-09 09:45:12 +02:00
09e611deb7
refactor: lift query schema generation up to caller
...
Do no longer scan chunks during query planning to determine the schema
(except for the lifetime jobs where we have a good reason to do so).
Instead pass the schema down to from whoever is triggering the query.
For real SQL queries, we then just use the the table-wide schemas
introduced in #1913 .
Apart from avoiding schema merges we now also don't crash any longer
when no chunks are left in the table (aka columns are present but all
rows are gone).
Fixes #1768 .
Fixes #1884 .
2021-07-09 09:24:21 +02:00
c37053ad46
Merge branch 'main' into cn/chunk-times
2021-07-08 20:58:54 +00:00
a2726c7e92
Merge branch 'main' into cn/kafka-read-metrics-and-e2e-tests
2021-07-08 20:40:19 +00:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
548c64539e
fix: Wrap lines at 100 chars
2021-07-08 16:39:33 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
c8126784a8
Merge branch 'main' into ntran/avoid_sort_in_scan
2021-07-08 20:22:18 +00:00
72928aab3d
refactor: Move ChunkLifecycleAction to the data_types crate ( #1939 )
2021-07-08 20:18:33 +00:00
dd3eff7748
refactor: Always use `row_count` for count of rows in system.* tables ( #1937 )
2021-07-08 19:28:11 +00:00
c6bf0a26f4
feat: Add metrics for when ingesting from the write buffer fails
...
So that we have some way of figuring out what might be going on.
2021-07-08 09:57:51 -04:00
80e1dcafe0
feat: Support reading from all Kafka partitions
...
When reading from the Kafka write buffer, subscribe to all partitions in
a topic and start from the smallest offset available, instead of
assuming there will only be 1 partition per topic.
2021-07-08 09:30:59 -04:00
c90ef7b14b
fix: Create one consumer group per server+database
...
This hasn't caused any problems for me yet, but seemed like a good idea
because we want to be sure we don't get any of Kafka's consumer
rebalancing if we have multiple partitions.
2021-07-08 09:28:34 -04:00
e5168936f5
feat: Better error messages through to gRPC API + e2e Kafka Read tests
2021-07-08 09:28:34 -04:00
c53ae41d57
fix: Remove unneeded Option from the reading mock
2021-07-08 09:28:34 -04:00
854c28c41a
feat: Stream messages from Kafka into the database
2021-07-08 09:28:34 -04:00
ee500f5bda
feat: Support configuring a write buffer for writing OR reading
2021-07-08 09:28:34 -04:00
63d26f6f3f
refactor: Rename KafkaBuffer to KafkaBufferProducer
2021-07-08 09:28:34 -04:00
e5de73133c
feat: Change write buffer connection rule to take either Writing or Reading connection info
...
A database on one IOx server can, exclusively:
- Not interact with Kafka at all
- Send writes to Kafka
- Read writes from Kafka
Notably, a database on a particular server will never write *and* read from Kafka at the same time.
2021-07-08 09:28:34 -04:00
fd4bcc2fa5
refactor: Rename the WriteBuffer trait to be WriteBufferWriting
2021-07-08 09:28:34 -04:00
83e50cfba4
refactor: Rename field to not contain the type
2021-07-08 09:28:34 -04:00
69e4786fc7
Merge branch 'main' into crepererum/str_arcs
2021-07-08 13:20:49 +00:00
18893e76e0
refactor: convert some table name and part. key String to Arcs
...
This has the (somewhat nice) side effect that it shrinks the in-mem
catalog a bit as well because nw `ParquetChunk` is a bit smaller making
the chunk stage enum smaller as well.
2021-07-08 14:34:28 +02:00
7ff8ae4ce5
refactor: tidy up sort key rep
2021-07-08 12:48:41 +01:00
f811bf1e5e
refactor: log compaction activity
2021-07-08 12:48:41 +01:00
33bc85ad18
feat: Infrastructure for persistence ( #1925 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:14:38 +00:00
7602bde850
chore: Update datafusion deps ( #1799 )
...
* chore: Update datafusion deps + rework code
* refactor: remove workaround as it has been contributed upstream
* fix: Update query/src/exec/split.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 10:58:32 +00:00
24056d7bfc
test: ensure that table schemas are recovered from pres. catalog
2021-07-08 10:01:42 +02:00
a746cd45c5
test: check for schema change errors
2021-07-08 09:51:49 +02:00
bd22dd38ea
docs: fix typos
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-08 09:18:09 +02:00
b528ac2b55
feat: store schemas per table
...
This way we can:
- check for schema matches even for writes going into different
partitions
- solve #1768 and #1884 in some future PR
Closes #1897 .
2021-07-08 09:18:09 +02:00
5ca9760c94
test: make partioning in DB tests consistent w/ DB rules
2021-07-08 09:18:09 +02:00
ed3ebdcbd2
refactor: use sync locks w/ better metrics
2021-07-08 09:18:09 +02:00
5936452895
feat: add infra to check table-wide schemas
2021-07-08 09:18:09 +02:00
5c722af0fa
fix: remove comments
2021-07-07 16:50:53 -04:00
d3c4f8c249
fix: store sort key correctly inthe schema. Update tests to reflect it
2021-07-07 15:55:23 -04:00
cc350bb1ea
fix: don't update last write time on failed writes
...
Fixes #1905
2021-07-07 14:50:03 -04:00
e6d995cbd8
chore: Update to Rust 1.53.0 ( #1922 )
...
* chore: Update to Rust 1.53.0
* fix: Update to latest clippy standards
* fix: bad refactor
* fix: Update escaping
* test: update test output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 18:02:03 +00:00
957c6245e3
docs: Note that rollover_partition is not automatically called ( #1910 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 12:14:31 +00:00
25e3a304ed
chore: Log partition rollover ( #1907 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 11:48:16 +00:00
8dfc3bb6bc
fix: Thanks Andrew for helping fic the compile problem and avoid using Arc<Mutex>
2021-07-06 18:05:59 -04:00
76789e5902
feat: store sotkey into the chunk schema of RUB
2021-07-06 17:00:35 -04:00
b6185982f7
refactor: make `ProviderBuilder` a build-time-checked builder
...
It's safer and also avoids cloning / copying state around.
2021-07-06 18:20:05 +02:00
4f5fe62428
feat: add DB name to lifecycle logs ( #1900 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-06 16:14:28 +00:00
09b7405b20
docs: spelling fixes
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-06 17:46:36 +02:00
3d644b63a1
feat: add `Replay` state to DB init
2021-07-06 14:24:39 +02:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
bcf43a3de5
Merge branch 'main' into crepererum/db_state_in_grpc
2021-07-05 07:21:48 +00:00
Andrew Lamb
Marco Neumann
kodiakhq[bot]
Carol (Nichols || Goulding)
Edd Robinson
Nga Tran
Paul Dix
Marko Mikulicic