c53ae41d57
fix: Remove unneeded Option from the reading mock
2021-07-08 09:28:34 -04:00
854c28c41a
feat: Stream messages from Kafka into the database
2021-07-08 09:28:34 -04:00
ee500f5bda
feat: Support configuring a write buffer for writing OR reading
2021-07-08 09:28:34 -04:00
63d26f6f3f
refactor: Rename KafkaBuffer to KafkaBufferProducer
2021-07-08 09:28:34 -04:00
e5de73133c
feat: Change write buffer connection rule to take either Writing or Reading connection info
...
A database on one IOx server can, exclusively:
- Not interact with Kafka at all
- Send writes to Kafka
- Read writes from Kafka
Notably, a database on a particular server will never write *and* read from Kafka at the same time.
2021-07-08 09:28:34 -04:00
fd4bcc2fa5
refactor: Rename the WriteBuffer trait to be WriteBufferWriting
2021-07-08 09:28:34 -04:00
83e50cfba4
refactor: Rename field to not contain the type
2021-07-08 09:28:34 -04:00
3a329d3705
Merge pull request #1931 from influxdata/crepererum/str_arcs
...
refactor: convert some table name and part. key String to Arcs
2021-07-08 13:28:04 +00:00
69e4786fc7
Merge branch 'main' into crepererum/str_arcs
2021-07-08 13:20:49 +00:00
18893e76e0
refactor: convert some table name and part. key String to Arcs
...
This has the (somewhat nice) side effect that it shrinks the in-mem
catalog a bit as well because nw `ParquetChunk` is a bit smaller making
the chunk stage enum smaller as well.
2021-07-08 14:34:28 +02:00
4c05d7ccfd
Merge pull request #1928 from influxdata/er/refactor/logging
...
refactor: log information for compaction results
2021-07-08 11:57:11 +00:00
7ff8ae4ce5
refactor: tidy up sort key rep
2021-07-08 12:48:41 +01:00
f811bf1e5e
refactor: log compaction activity
2021-07-08 12:48:41 +01:00
86372c8818
refactor: remove unused method
2021-07-08 12:48:41 +01:00
33bc85ad18
feat: Infrastructure for persistence ( #1925 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:14:38 +00:00
f670224ea1
chore: Reduce output spew during query tests ( #1926 )
...
* chore: Reduce output spew during query tests
* docs: Update query_tests/src/runner.rs
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:06:24 +00:00
7602bde850
chore: Update datafusion deps ( #1799 )
...
* chore: Update datafusion deps + rework code
* refactor: remove workaround as it has been contributed upstream
* fix: Update query/src/exec/split.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 10:58:32 +00:00
7b18c29d32
Merge pull request #1915 from influxdata/fixnonelinger
...
fix: Ensure mutable_linger_seconds can not be None
2021-07-08 09:33:15 +00:00
7059f16b9e
refactor: Turn mutable_linger_seconds into a non-optional ( #1917 )
2021-07-08 11:25:57 +02:00
0ec11eb907
fix: Handle None mutable_linger_seconds in can_move
...
This will be followed by a cleanup PR that will no longer make mutable_linger_seconds an option
and move the defaulting to the configuration layer.
Closes #1899
2021-07-08 11:02:09 +02:00
ea2bbab0f1
Merge pull request #1913 from influxdata/crepererum/issue1897
...
feat: store schemas per table, enforce schema on write
2021-07-08 08:08:59 +00:00
24056d7bfc
test: ensure that table schemas are recovered from pres. catalog
2021-07-08 10:01:42 +02:00
a746cd45c5
test: check for schema change errors
2021-07-08 09:51:49 +02:00
bd22dd38ea
docs: fix typos
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-08 09:18:09 +02:00
b528ac2b55
feat: store schemas per table
...
This way we can:
- check for schema matches even for writes going into different
partitions
- solve #1768 and #1884 in some future PR
Closes #1897 .
2021-07-08 09:18:09 +02:00
5ca9760c94
test: make partioning in DB tests consistent w/ DB rules
2021-07-08 09:18:09 +02:00
ed3ebdcbd2
refactor: use sync locks w/ better metrics
2021-07-08 09:18:09 +02:00
5936452895
feat: add infra to check table-wide schemas
2021-07-08 09:18:09 +02:00
315217362f
feat: add TableBatch -> Schema conversion
2021-07-08 09:18:09 +02:00
4e5d5c8c4c
Merge pull request #1924 from influxdata/pd-fix-last-write
...
fix: don't update last write time on failed writes
2021-07-07 19:07:37 +00:00
cc350bb1ea
fix: don't update last write time on failed writes
...
Fixes #1905
2021-07-07 14:50:03 -04:00
ac1c7a5e07
Merge pull request #1923 from influxdata/cn/one-more-fb-fix
...
fix: Outdated reference to a moved file in docs
2021-07-07 18:11:47 +00:00
94f32e357f
Merge branch 'main' into cn/one-more-fb-fix
2021-07-07 18:02:39 +00:00
e6d995cbd8
chore: Update to Rust 1.53.0 ( #1922 )
...
* chore: Update to Rust 1.53.0
* fix: Update to latest clippy standards
* fix: bad refactor
* fix: Update escaping
* test: update test output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 18:02:03 +00:00
479b15c2a3
fix: Outdated reference to a moved file in docs
2021-07-07 13:55:35 -04:00
090b0aba11
refactor: remove unused `mutable_size_threshold` lifecycle setting ( #1909 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 17:03:15 +00:00
d4f4af9068
fix: fix flatbuffers regeneration code ( #1898 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 14:42:47 +00:00
957c6245e3
docs: Note that rollover_partition is not automatically called ( #1910 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 12:14:31 +00:00
c63e2fe605
chore: Add debug logs to maybe_compact_chunks ( #1911 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 12:06:26 +00:00
25e3a304ed
chore: Log partition rollover ( #1907 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 11:48:16 +00:00
3f0cd378f1
Merge pull request #1908 from influxdata/fixcanmove
...
chore: Add some more tests for can_move
2021-07-07 11:18:33 +00:00
17b82ebcd6
chore: Add some more tests for can_move
2021-07-07 13:03:38 +02:00
d075aaa0be
Merge pull request #1904 from influxdata/lifelogs
...
fix: Increase log level for important lifecycle events
2021-07-07 10:21:53 +00:00
41f49db3c1
fix: Increase log level for important lifecycle events
...
This can possibly help us better understand #1899 .
The practical reason for increasing the log level for those
events is that we cannot currently ingest debug logs in our log aggregation system,
but it currently takes 7h to reproduce #1899 which means we don't have access to debug logs
from `kubectl logs` to help us troubleshoot.
That said, I do think that these logs are not debug logs but legit lifecycle information.
In particular if a log says "unexpected ..." I think it may be better regarded as an error/warning.
(If it becomes too verbose, it means its either a bug or not that unexpected).
2021-07-07 10:57:56 +02:00
a50c1e75e5
Merge pull request #1901 from influxdata/crepererum/builder_pattern_fix
...
refactor: rework builder pattern
2021-07-07 07:23:32 +00:00
b6185982f7
refactor: make `ProviderBuilder` a build-time-checked builder
...
It's safer and also avoids cloning / copying state around.
2021-07-06 18:20:05 +02:00
4172d7946c
refactor: make `SchemaMerger` self-consuming
...
The error handling in `merge` was incomplete, aka it could leave the
merger in a half-modified state in case of an error. That's generally a
bad idea and can lead to ugly bugs. Also the "builder" pattern that is
used here usually consumes itself (and provides a clone impl), so it is
easier to reason about modifications. So this commit just changes it to
self-consuming builder.
A nice side effect of the new pattern is also that it is build-time
checked and does not contain a runtime assert any longer.
2021-07-06 18:20:05 +02:00
4f5fe62428
feat: add DB name to lifecycle logs ( #1900 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-06 16:14:28 +00:00
c7e3f05f76
Merge pull request #1895 from influxdata/crepererum/catalog_replay2
...
feat: add `Replay` state to DB init
2021-07-06 15:55:19 +00:00
004ce63ec5
Merge branch 'main' into crepererum/catalog_replay2
2021-07-06 15:47:05 +00:00
Carol (Nichols || Goulding)
kodiakhq[bot]
Marco Neumann
Edd Robinson
Andrew Lamb
Marko Mikulicic
Paul Dix