f57ba6afdb
fix: use fixed-size timestamps for parquet metadata ( #2032 )
...
This fixes flaky tests that rely on predictable files sizes.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-16 13:14:02 +00:00
2498642c00
fix: `persist_partition` docstring
2021-07-16 12:46:07 +02:00
1ef2bc1887
refactor: `Db::{write_chunk_to_object_store => Db::persist_partition}`
...
The previous method allowed to persist any chunk -- even ones that
should not be persisted yet and w/o any order of peristence. That will
break our persistence windows. So instead offer a sane higher-level
interface that can trigger persistence of a partition within the
boundaries of the lifecycle rules. This needs some adjustments for our
test suite.
2021-07-16 12:07:58 +02:00
9683d91f32
refactor: adjust to upstream changes
2021-07-16 11:45:34 +02:00
2b0a4bbe0a
feat: persist real (non-fake) part.+DB checkpoints
2021-07-16 11:45:34 +02:00
8276511bd3
feat: allow to construct partition checkpoint from partition
2021-07-16 11:45:34 +02:00
a9ea8e9ced
docs: add docstring to some `Partition` methods
2021-07-16 11:45:34 +02:00
71b5030fc0
refactor: remove unused `LockableChunk::write_to_object_store`
2021-07-16 11:45:34 +02:00
00b89cd751
fix: freeze chunks in write path ( #2021 ) ( #2022 )
...
* fix: freeze chunks in write path (#2021 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-16 08:51:37 +00:00
50aa1f857d
Merge branch 'main' into ntran/refactor_use_sort_key
2021-07-15 21:17:26 +00:00
76d9b8f7cc
Merge branch 'main' into debugkafka
2021-07-15 21:07:35 +00:00
d5dcb40438
refactor: track future execution ( #2014 )
...
* refactor: track future execution
* refactor: update server/src/db/lifecycle/compact.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 20:58:52 +00:00
06399e88e0
chore: Add some debug logs to write buffer
2021-07-15 22:18:03 +02:00
cfe0bfa88b
refactor: address review comments and add useful log info to catch resort
2021-07-15 15:39:12 -04:00
3fd6430fb6
fix: rename `estimated_bytes` to `memory_bytes` and expose `object_store_bytes` in ChunkSummary and system.chunks ( #2017 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 16:00:24 +00:00
b5428e53a5
refactor: write buffer testing + better mocking
...
This refactors the write buffer a bit for:
- **Testing:** Add generic tests for the Kafka and the mocking
implementation. The same interface can be used easily add new
implementations (e.g. via Redis, filesystem, ...).
- **Partition on Write:** The caller of the writer operation must now
specify the partition/sequencer ID. The implicit partitioning of the
Kafka writer would have lead to broken data since we must never spill
entries w/ the same primary key over multiple partitions. At the
moment we will only use partition 0 but we can easily implement
better logic in the future.
- **Improved Mocking:** The mocked implementation now simulates a system
that feels more real. Especially the handling around multiple streams
and "write while read" has been improved. This will be helpful for
testing and for new features like seeking (during replay). A solid
realistic mock also helps us to ensure that the tests using the mock
do not rely on unrealistic behavior too much.
2021-07-15 17:20:45 +02:00
d71f38f27c
feat: compute PartitionCheckpoint from PersistenceWindows ( #2011 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-15 12:17:23 +00:00
0c86d1dccf
feat: Record parquet bytes size in catalog / parquet_file ( #2006 )
...
* feat: Store object store size in parquet_file
* fix: update TRANSACTION_VERSION to 8
* refactor: rename os_bytes --> file_size_bytes
2021-07-15 12:07:11 +00:00
4741483f72
docs: explain why we update memory metrics when lifecycle action is cleared
2021-07-15 12:07:56 +02:00
924b0db542
fix: account for memory size in drop lifecycle action
2021-07-15 12:07:56 +02:00
cccdd8a43f
fix: correct code comment
2021-07-15 12:07:56 +02:00
77a9191a11
fix: chunk dropping over lifecycle policy should also respect the preserved catalog
2021-07-15 12:07:56 +02:00
71cb15f017
refactor: use lifecycle action to drop chunks
...
This avoids holding partition locks while the preserved catalog IO is
done.
2021-07-15 12:07:56 +02:00
e570c66697
feat: add "dropping" chunk lifecycle action
2021-07-15 12:07:56 +02:00
68e20779a2
test: add test for clearing lifecycle actions from chunks
2021-07-15 12:07:56 +02:00
d89fca00be
feat: persist "drop chunk"
2021-07-15 12:07:56 +02:00
3e0d1eb560
refactor: introduce PartitionAddr ( #2010 )
2021-07-15 10:01:33 +00:00
0b1f2b1fd0
chore: merge main to branch
2021-07-14 16:17:14 -04:00
552e3fb691
fix: Padd stats compute deterministic order of sort key and update tests that got changed by the use of sort key
2021-07-14 14:06:41 -04:00
833debd5b5
Merge branch 'main' into cn/exploration
2021-07-14 17:30:55 +00:00
cbeeb97cff
feat: flush open window on persist ( #1985 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 16:58:20 +00:00
1d00fa2fd8
refactor: track memory metrics in catalog ( #1995 )
...
* refactor: track memory metrics in catalog
* chore: update comment
2021-07-14 16:23:00 +00:00
8070065e2f
fix: Change RUB chunk table_summaries to table_summary
...
Because chunks now have only one table.
Connects to #1718 , #1613 , #1295
2021-07-14 11:18:02 -04:00
649b467adb
fix: CatalogChunk no longer needs to record a write when created from a MUB chunk
2021-07-14 10:28:12 -04:00
7ccbab8c90
feat: Make a TableSummaryAndTimes to use to slowly replace TableSummary
...
And use TableSummaryAndTimes with the mutable buffer chunks when turning
them into catalog chunks.
It's proving too big to switch over everything using TableSummary at
once, so this will let us switch over more incrementally.
2021-07-14 10:28:12 -04:00
4dedb657f2
Merge branch 'main' into alamb/go_go_go_go
2021-07-14 14:04:13 +01:00
f1c1620c84
feat: make persistence windows interface harder to use incorrectly ( #1977 )
...
* feat: make persistence windows interface harder to use incorrectly
* chore: review feedback
* chore: update comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-14 13:03:18 +00:00
0e5276ed20
Merge branch 'main' into alamb/go_go_go_go
2021-07-14 13:56:35 +01:00
9cb9ae0874
chore: move write buffer into its own crate
2021-07-14 14:09:18 +02:00
d427fed9dc
fix: Remove bad max.request.size config param
2021-07-14 13:54:18 +02:00
8fd0df04f2
feat: continue buidling and using sort_key if available
2021-07-13 16:25:58 -04:00
781c4fa666
fix: update server tests
2021-07-13 15:44:57 -04:00
239c931f26
fix: Raise max message to 10M
...
And log message size on kafka write error.
Turns out the kafka partition message size limit default is 1MB, but also the
client side "max request size" default is also 1MB.
The error message we get from our kafka client is misleading: it says
```
KafkaError (Message production error: MessageSizeTooLarge (Broker: Message size too large)) }
```
which to my mind it seemed like if ("Broker:") the broker said "Message size too large".
That was a lie; I killed the broker and the client kept saying the same error message which means
it didn't even try to send the message out.
TODO: make this a proper parameter. (but let's unblock)
2021-07-13 17:47:36 +02:00
6a09678f34
Merge branch 'main' into crepererum/update_deps
2021-07-13 14:18:57 +00:00
6c8b2b4fa7
feat: add integration test of compaction freezing ( #1938 ) ( #1975 )
...
* feat: add integration test of compaction freezing (#1938 )
* chore: update server/src/db/lifecycle/compact.rs
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-13 14:11:10 +00:00
157a0cc98c
chore: update flatbuffers to 2.0
2021-07-13 15:44:45 +02:00
bf20641d78
chore: Log whether the write buffer is enabled
2021-07-13 14:15:52 +02:00
5a0caeab44
feat: skip over fully persisted partitions ( #1962 ) ( #1973 )
...
* feat: skip over fully persisted partitions (#1962 )
* chore: review feedback
2021-07-13 10:40:45 +00:00
d35b74c226
fix: Fix doc build warnings ( #1945 )
...
* fix: Fix doc build warnings
* refactor: add deny bare_urls to crates
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-13 08:03:42 +00:00
708aebaeb3
fix: freeze open chunk when compacting ( #1971 )
...
Closes #1938 . Unfortunately, this contains only a unit test to ensure that open chunks are frozen when set_compacting is called. It would be better to have a more end-to-end integration test that ensurest his behavior, but I've confirmed by hand (with some sleeps and a hacked up end-to-end test) that this fixes it.
2021-07-13 07:44:02 +00:00
5418a1fe6b
refactor: remove unused comments
2021-07-12 18:14:38 -04:00
23895e6673
feat: Using sort_key to avoid resorts
2021-07-12 18:08:45 -04:00
6764a2d68e
fix: Write Buffer errors are known, not UnknownDatabaseErrors
...
Fixes #1956 .
2021-07-12 11:21:31 -04:00
3bd7486016
test: Rename a test type alias to not shadow super::Error
2021-07-12 10:46:29 -04:00
670826daf9
refactor: make object_store construction interface consistent ( #1944 )
...
* refactor: make object_store construction interface consistent
* fix: benchmarks
* fix: doc build
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-12 12:56:36 +00:00
9534220035
feat: Add any lifecycle_action to system.chunks and API ( #1947 )
2021-07-09 17:38:29 +00:00
7af560aa99
feat: Persist lifecycle action ( #1888 )
...
* feat: add split and persist operation
* docs: Improve doc strings
* refactor: use for loop rather than map
* refactor: Make it clear that the lifecycle policy picks the split timestamp
* fix: race condition
* docs: improve comments
* fix: logical merge conflict
* fix: clippy
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-07-09 13:21:46 +00:00
1a79bf7e99
refactor: Make aws/azure/gcs optional features and stop compiling 100 dependencies during dev ( #1933 )
...
* feat: make aws, gcp, azure dependencies optional
* fix: only run object store tests if the features are enabled
* fix: clean up testing
* fix: rename step
* fix: add to list of jobs
* fix: remove test with object store
* fix: review comments
2021-07-09 11:38:30 +00:00
3cb8f297b1
refactor: encapsulate the ObjectStore implementations in the object store crate ( #1932 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-09 10:38:32 +00:00
bc958e2ff0
refactor: use Arcs to pass schemas around
2021-07-09 09:45:12 +02:00
09e611deb7
refactor: lift query schema generation up to caller
...
Do no longer scan chunks during query planning to determine the schema
(except for the lifetime jobs where we have a good reason to do so).
Instead pass the schema down to from whoever is triggering the query.
For real SQL queries, we then just use the the table-wide schemas
introduced in #1913 .
Apart from avoiding schema merges we now also don't crash any longer
when no chunks are left in the table (aka columns are present but all
rows are gone).
Fixes #1768 .
Fixes #1884 .
2021-07-09 09:24:21 +02:00
c37053ad46
Merge branch 'main' into cn/chunk-times
2021-07-08 20:58:54 +00:00
a2726c7e92
Merge branch 'main' into cn/kafka-read-metrics-and-e2e-tests
2021-07-08 20:40:19 +00:00
22495dd355
fix: Take a TableBatch in the MBChunk constructor
...
Thus ensuring all MBChunks will have data in them.
2021-07-08 16:39:35 -04:00
548c64539e
fix: Wrap lines at 100 chars
2021-07-08 16:39:33 -04:00
74c0a6cb00
fix: Arrange use statements so rustfmt can manage their order
2021-07-08 16:39:02 -04:00
c8126784a8
Merge branch 'main' into ntran/avoid_sort_in_scan
2021-07-08 20:22:18 +00:00
72928aab3d
refactor: Move ChunkLifecycleAction to the data_types crate ( #1939 )
2021-07-08 20:18:33 +00:00
dd3eff7748
refactor: Always use `row_count` for count of rows in system.* tables ( #1937 )
2021-07-08 19:28:11 +00:00
c6bf0a26f4
feat: Add metrics for when ingesting from the write buffer fails
...
So that we have some way of figuring out what might be going on.
2021-07-08 09:57:51 -04:00
80e1dcafe0
feat: Support reading from all Kafka partitions
...
When reading from the Kafka write buffer, subscribe to all partitions in
a topic and start from the smallest offset available, instead of
assuming there will only be 1 partition per topic.
2021-07-08 09:30:59 -04:00
c90ef7b14b
fix: Create one consumer group per server+database
...
This hasn't caused any problems for me yet, but seemed like a good idea
because we want to be sure we don't get any of Kafka's consumer
rebalancing if we have multiple partitions.
2021-07-08 09:28:34 -04:00
e5168936f5
feat: Better error messages through to gRPC API + e2e Kafka Read tests
2021-07-08 09:28:34 -04:00
c53ae41d57
fix: Remove unneeded Option from the reading mock
2021-07-08 09:28:34 -04:00
854c28c41a
feat: Stream messages from Kafka into the database
2021-07-08 09:28:34 -04:00
ee500f5bda
feat: Support configuring a write buffer for writing OR reading
2021-07-08 09:28:34 -04:00
63d26f6f3f
refactor: Rename KafkaBuffer to KafkaBufferProducer
2021-07-08 09:28:34 -04:00
e5de73133c
feat: Change write buffer connection rule to take either Writing or Reading connection info
...
A database on one IOx server can, exclusively:
- Not interact with Kafka at all
- Send writes to Kafka
- Read writes from Kafka
Notably, a database on a particular server will never write *and* read from Kafka at the same time.
2021-07-08 09:28:34 -04:00
fd4bcc2fa5
refactor: Rename the WriteBuffer trait to be WriteBufferWriting
2021-07-08 09:28:34 -04:00
83e50cfba4
refactor: Rename field to not contain the type
2021-07-08 09:28:34 -04:00
69e4786fc7
Merge branch 'main' into crepererum/str_arcs
2021-07-08 13:20:49 +00:00
18893e76e0
refactor: convert some table name and part. key String to Arcs
...
This has the (somewhat nice) side effect that it shrinks the in-mem
catalog a bit as well because nw `ParquetChunk` is a bit smaller making
the chunk stage enum smaller as well.
2021-07-08 14:34:28 +02:00
7ff8ae4ce5
refactor: tidy up sort key rep
2021-07-08 12:48:41 +01:00
f811bf1e5e
refactor: log compaction activity
2021-07-08 12:48:41 +01:00
33bc85ad18
feat: Infrastructure for persistence ( #1925 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:14:38 +00:00
7602bde850
chore: Update datafusion deps ( #1799 )
...
* chore: Update datafusion deps + rework code
* refactor: remove workaround as it has been contributed upstream
* fix: Update query/src/exec/split.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 10:58:32 +00:00
24056d7bfc
test: ensure that table schemas are recovered from pres. catalog
2021-07-08 10:01:42 +02:00
a746cd45c5
test: check for schema change errors
2021-07-08 09:51:49 +02:00
bd22dd38ea
docs: fix typos
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-08 09:18:09 +02:00
b528ac2b55
feat: store schemas per table
...
This way we can:
- check for schema matches even for writes going into different
partitions
- solve #1768 and #1884 in some future PR
Closes #1897 .
2021-07-08 09:18:09 +02:00
5ca9760c94
test: make partioning in DB tests consistent w/ DB rules
2021-07-08 09:18:09 +02:00
ed3ebdcbd2
refactor: use sync locks w/ better metrics
2021-07-08 09:18:09 +02:00
5936452895
feat: add infra to check table-wide schemas
2021-07-08 09:18:09 +02:00
5c722af0fa
fix: remove comments
2021-07-07 16:50:53 -04:00
d3c4f8c249
fix: store sort key correctly inthe schema. Update tests to reflect it
2021-07-07 15:55:23 -04:00
cc350bb1ea
fix: don't update last write time on failed writes
...
Fixes #1905
2021-07-07 14:50:03 -04:00
e6d995cbd8
chore: Update to Rust 1.53.0 ( #1922 )
...
* chore: Update to Rust 1.53.0
* fix: Update to latest clippy standards
* fix: bad refactor
* fix: Update escaping
* test: update test output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 18:02:03 +00:00
957c6245e3
docs: Note that rollover_partition is not automatically called ( #1910 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 12:14:31 +00:00
25e3a304ed
chore: Log partition rollover ( #1907 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 11:48:16 +00:00
8dfc3bb6bc
fix: Thanks Andrew for helping fic the compile problem and avoid using Arc<Mutex>
2021-07-06 18:05:59 -04:00
76789e5902
feat: store sotkey into the chunk schema of RUB
2021-07-06 17:00:35 -04:00
b6185982f7
refactor: make `ProviderBuilder` a build-time-checked builder
...
It's safer and also avoids cloning / copying state around.
2021-07-06 18:20:05 +02:00
4f5fe62428
feat: add DB name to lifecycle logs ( #1900 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-06 16:14:28 +00:00
09b7405b20
docs: spelling fixes
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-07-06 17:46:36 +02:00
3d644b63a1
feat: add `Replay` state to DB init
2021-07-06 14:24:39 +02:00
4ca2d3e148
chore: move persistence windows related code into own crate
...
The entire persistence windows data structures (including the
checkpoints) have nothing to do with the mutable buffer per se. So lets
move them into their own crate. This also makes `parquet_file` not
longer depend on `mutable_buffer`.
2021-07-05 10:23:58 +02:00
cdab1bed05
feat: persist part+db checkpoint in parquets and catalog
...
This will be required for replay on server startup.
2021-07-05 09:42:46 +02:00
bcf43a3de5
Merge branch 'main' into crepererum/db_state_in_grpc
2021-07-05 07:21:48 +00:00
405a6a691b
feat: intial implementation of #1886 : avoid resort if appropriate
2021-07-02 17:57:48 -04:00
b4534883fe
refactor: remove table name from upsert_table ( #1882 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-02 15:22:41 +00:00
54fbb60740
feat: expose DB state in gRPC interface
2021-07-02 11:24:36 +02:00
404da38d6f
Merge branch 'main' into pd-remove-mb-size-limit-checks
2021-07-01 20:01:32 +00:00
5b00bc69e6
refactor: use Arc<Db> in lifecycle actions ( #1873 )
...
* refactor: use Arc<Db> in lifecycle actions
* chore: review feedback
2021-07-01 19:56:33 +00:00
61917c107f
chore: add test for can_move on row count
2021-07-01 15:49:44 -04:00
91f5478012
feat: remove MUB size threshold
...
Removes the MUB chunk close based on size. Also add a check in lifecycle policy to move if the MUB chunk crosses a default row count threshold.
2021-07-01 14:58:29 -04:00
56c8c8d428
feat: Use separate executor for queries and compactions/moves ( #1870 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:47:50 +00:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
07826306ed
fix: Always deduplicate data prior to insertion into the ReadBuffer ( #1863 )
...
* fix: mark ReadBuffer as always deduplicated
* fix: Use compact plans during merge
* docs: Update server/src/db/chunk.rs
Co-authored-by: Nga Tran <ntran@influxdata.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Nga Tran <ntran@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:23:37 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
e1e3163752
refactor: rework DB init state machine
...
Since adding new features like "sequencer replay" or init retries would
make the current code too complex, a refactor is required:
Config:
The config struct now holds a `DatabaseState` which is a simple linear
state machine representing the different stages of the database init.
Init:
The init module now has a fixpoint-loop which looks at the state,
decides what to do based on it and repeats until either the DB is
initialized or an error occured. This also makes it easier to continue
the init process "in the middle", e.g. when the preserved catalog is
broken or the sequencer (e.g. Kafka) could not be reached.
2021-07-01 13:47:51 +02:00
cfa06e1497
chore: Add query tests for compacted chunks ( #1861 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 20:59:29 +00:00
99a15cd452
refactor: single lifecycle error enumeration ( #1859 )
...
* refactor: single lifecycle error enumeration
* fix: fmt
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-06-30 18:35:57 +00:00
817a480cde
refactor: move lifecycle implementations out of db.rs and into their own modules ( #1858 )
...
* refactor: move lifecycle implementations out of db.rs and into their own modules
* fix: clippy
2021-06-30 17:24:04 +00:00
9e1723620c
refactor: rename load_chunk_to_read_buffer to move_chunk_to_read_buffer ( #1857 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 16:53:18 +00:00
043890369f
refactor: make `MinMaxSequence` safer to use
2021-06-30 16:37:48 +02:00
983062f6fa
Merge branch 'main' into crepererum/no_catalog_on_db_creation
2021-06-30 10:04:00 +00:00
2e430ac7f0
refactor: remove table name from read_filter schema
2021-06-30 09:50:53 +01:00
62f274cc1b
refactor: remove table name from column_values
2021-06-30 09:46:54 +01:00
5737c9d962
refactor: remove table name from column_names
2021-06-30 09:43:41 +01:00
c4e054f909
feat: do NOT load preserved catalogs on late DB creation
...
When a DB is created AFTER the server is initialized, then we can assume
it is a new DB (because the rules file did not exist beforehand). We
shall treat it as a new DB with no data and should not try to load some
leftover / stale / whatever preserved catalog for it. How this catalog
came into existence we do not know and it was certainly not properly
managed by IOx. So we error if there is a catalog.
Furthermore the old implementation was kinda broken since it loaded the
perserved catalog "in-sync" with the gRPC call that issued the DB
creation (we only have a delayed init concept for DBs that are loaded on
instance startup). In production that would very likely provoke nasty
timeouts.
On top of that this new behavior will also be somewhat more sane when we
think about sequencer (e.g. Kafka) replays. We certainly do not wanna do
any replays for newly created DBs.
TLDR: New behavior for DBs created via gRPC is "new empty DB". This does
NOT affect DBs loaded on instance startup (aka existing DBs).
2021-06-30 10:12:38 +02:00
58310abfee
refactor: de-duplicate code in `server::db::load`
2021-06-30 10:08:25 +02:00
9d10ac9f6a
refactor: write parquet files w/o holding the transaction lock
...
This allows to prepare writes per-tableXpartition before entering the
database-exclusive section that deals with catalog transactions.
Closes #1821 .
2021-06-29 14:23:06 +02:00
3ebb6a3037
refactor: do not capture txn-specific information in parquet files
...
This helps with #1821 .
2021-06-29 14:22:36 +02:00
a7198ea78b
refactor: use satisfies_predicate in apply_predicate
2021-06-29 11:58:28 +01:00
eda9532eb2
Merge branch 'main' into crepererum/issue1821-cleanup-lock
2021-06-29 10:48:43 +00:00
3ee96c4618
fix: Do not sequence local writes (avoid panic under load) ( #1826 )
...
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
2cd5ce98be
refactor: do not pass locks around for catalog cleanup
2021-06-29 10:21:41 +02:00
730a23faa3
refactor: improve locking around the parquet file cleanup
...
Instead of (ab)using the transaction lock to prevent the cleanup job
from removing just-written parquet files, use a dedicated lock. This
will later allow us to write parquet files before starting a transaction
(i.e. w/o holding the transaction lock).
This will help with #1821 .
2021-06-29 10:20:03 +02:00
12ae9b012a
refactor: clarify intent of
2021-06-28 17:39:48 +01:00
0f7c47d10e
fix: Limit the number of errors per sequenced entry we'll collect
2021-06-28 09:29:17 -04:00
1e171e2e9a
refactor: Organize `use` statements and let rustfmt manage order
2021-06-28 09:29:15 -04:00
f3a3a9b267
fix: Try to write all partition_writes even if one fails, collect all errors and report at the end
2021-06-28 09:24:23 -04:00
4d2954ec1d
test: Write a failing tests for partition_writes being ignored after a failure
2021-06-28 09:24:23 -04:00
65e65412cc
refactor: move catalog loading code into its own module
2021-06-28 12:46:25 +02:00
de236c5a6f
feat: update persistence windows to support late arrival less than 30 seconds
2021-06-25 15:34:11 -04:00
435b4b6a94
feat: add persistence windows to partition and update on write
...
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
2021-06-25 15:34:11 -04:00
3046b1692c
chore: include table name in compaction log ( #1805 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-25 15:20:44 +00:00
79446d45be
feat: Implement split_plans ( #1794 )
...
* feat: implement split plan / planner
* fix: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: resolve merge conflicts
* fix: add values to panic
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-06-24 18:38:00 +00:00
297fc12db8
feat: compact chunks ( #1776 )
...
* feat: compact chunks
* chore: review feedback
* chore: clippy lints
* chore: document sort key algorithm
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-24 16:49:10 +00:00
c0c1c3fd8e
refactor: Extract a struct to hold all the arguments needed to make a Db
2021-06-23 16:31:38 -04:00
f903b6eca8
fix: Create WriteBuffer outside of commit_db so committing can't fail
2021-06-23 13:56:50 -04:00
51e72c8821
refactor: Extract a function for creating a write buffer from database rules
2021-06-23 13:33:29 -04:00
57aee2f770
fix: Remove TODO that's a TODON'T
2021-06-23 13:18:28 -04:00
6ec3c03b0a
fix: Handle failure to create a Kafka producer rather than panicking
2021-06-23 10:51:23 -04:00
c66f9e5aeb
feat: Write entries to Kafka when configured as the write buffer
2021-06-23 10:48:18 -04:00
08f0696890
refactor: Extract a type alias for the trait's error type
2021-06-23 10:48:18 -04:00
250b9362a6
fix: Pass the database to the KafkaBuffer to use as the topic
2021-06-23 10:48:18 -04:00
93881da016
feat: Make Write Buffer store_entry async
...
In preparation for the Kafka write buffer implementation needing to call
async functions.
2021-06-23 10:48:18 -04:00
59993e8b8f
Merge branch 'main' into crepererum/issue1623
2021-06-23 12:40:05 +00:00
c395409b51
feat: include UUIDv4 into parquet file names
...
Change schema from
```text
<server_id>/<db_name>/data/<part_key>/<chunk_id>/<table_name>.parquet
```
to
```text
<server_id>/<db_name>/data/<table_name>/<part_key>/<chunk_id>.<uuid>.parquet
```
So parquet files will NEVER be overwritten. This is especially helpful
when dealing with old catalog leftovers (i.e. a parquet file that
belonged to an old but wiped catalog). It also simplifies the reasoning
about file references in the future and follows what other dataset
formats are usually doing (i.e. never replace files).
Also use `ChunkAddr` where it makes sense.
2021-06-23 14:30:28 +02:00
70817a474c
Merge branch 'main' into crepererum/issue1740-d
2021-06-23 12:29:54 +00:00
5cd911c74a
fix: correct row count for object store chunks ( #1789 )
2021-06-23 12:06:49 +00:00
d94a9ea94a
Merge branch 'main' into crepererum/better_served_uninit_error
2021-06-23 08:54:48 +00:00
cf55df68b5
refactor: remove some `Arc`s around the in-mem catalog
...
This is for #1740 .
2021-06-23 10:51:22 +02:00
39eac62d5d
fix: improve "server not initialized" error
...
We've reported "databases not loaded" which is a bit confusing for
router nodes, so change the description to "server not initialized".
2021-06-23 10:47:51 +02:00
d2be641864
refactor: make checkpointing easier to use
...
Don't mix commit+checkpoint in a single call so that the caller has to
reason about the error type and which of the two operations has failed.
Splitting it also makes it easier to create the correct checkpoint data.
2021-06-23 10:25:05 +02:00
4a961694ec
refactor: make caller sync mem<>OS view during catalog transactions
...
This is for #1740 . Greatly simplifies the integration of the persisted
catalog into the DB.
2021-06-23 10:25:05 +02:00
c3dbe4c571
Merge branch 'main' into crepererum/fix_auto_wipe
2021-06-22 13:50:53 +00:00
a98b10745f
fix: auto-wipe should still be enabled
...
Auto-wipe broken catalogs should be enabled until #1522 is closed.
2021-06-22 15:45:32 +02:00
b77bff449b
Merge branch 'main' into crepererum/issue1740-b
2021-06-22 13:27:26 +00:00
01b0fdabb7
feat: make lifecycle partition-aware ( #1767 )
...
* feat: make lifecycle partition-aware
* chore: further docs
* chore: rename to maybe_free_memory
* chore: fix logical conflicts
* chore: ensure only drops unpersisted chunks
* chore: clippy lints
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-22 09:24:15 +00:00
d1db0dfaeb
refactor: remove type parameter from preserved catalog
...
For #1740 .
2021-06-22 10:53:10 +02:00
799e2caa34
Merge branch 'main' into crepererum/issue1740-a
2021-06-22 07:19:27 +00:00
5362c7c924
feat: enable query deduplication ( #1762 )
2021-06-21 18:49:04 +00:00
ff60627500
refactor: make preserved catalog NOT own the in-mem catalog
...
Works towards #1740 .
2021-06-21 18:39:43 +02:00
881729bd23
refactor: make caller responsible to create checkpoint data
...
This decouples the in-mem and preserved catalog a bit and works
towards #1740 .
2021-06-21 18:33:23 +02:00
7e3df17896
test: update benchmarks
2021-06-21 15:29:23 +01:00
ac54320821
refactor: update server with new chunk API
2021-06-21 15:12:17 +01:00
ea04ce40dc
feat: transactional lifecycle API ( #1753 )
...
* feat: transactional lifecycle API
* chore: remove redundant upgrade
* feat: lifecycle error propagation
* chore: add usage doctest
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-21 13:09:53 +00:00
0d7c3ff279
docs: fix typos
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-06-21 13:18:20 +02:00
4d3432a1e0
docs: improve `server::config` docs
2021-06-21 10:06:50 +02:00
29bbc9a384
refactor: recoverable DB init
...
- store read and parsed DB rules even the the catalog is broken
- allow wiping the catalog for DBs w/ init failures
- try to bring the DB back online after successful wipes
Note that this does yet allow to update rules for broken DBs or to fix
DBs w/ broken rule files. However this can be implemented easily on top
of this.
2021-06-21 09:31:23 +02:00
d17b5710a8
feat: add server functionality to wipe preserved catalogs
2021-06-21 09:31:23 +02:00
aba973a6e1
refactor: make catalog `wipe` a freestanding function
...
It does not interact with the `CatalogState` so users can call this
function without that type.
2021-06-21 09:31:23 +02:00
258a6b1956
chore: remove more dead code ( #1760 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-18 21:28:22 +00:00
1d8469951f
Merge branch 'main' into smaller-cache
2021-06-18 18:50:10 +00:00
de67bd3efe
refactor: Remove PartitionChunk::table_schema ( #1756 )
...
* refactor: Remove PartitionChunk::table_schema
* docs: update comments
2021-06-18 16:13:16 +00:00
9beeca3e7c
refactor: Unify schema handling in query crate ( #1755 )
...
* refactor: Unify schema handling in query crate
* fix: doclink
2021-06-18 14:10:57 +00:00
1c13d676b4
refactor: Rename query::PartitionChunk --> query::QueryChunk ( #1754 )
2021-06-18 13:24:09 +00:00
b612c3af4e
chore: Switch to smaller cache dep
2021-06-18 09:43:28 +02:00
ec43a87909
chore: Update itertools deps ( #1750 )
2021-06-17 17:56:44 +00:00
f6dbc8d6f2
refactor: add ChunkAddr to describe location of chunk in catalog ( #1745 )
...
* refactor: add ChunkPath to describe location of chunk in catalog
* refactor: rename ChunkPath to ChunkAddr
* chore: further renames
* chore: even more renames
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-17 12:04:37 +00:00
87b2a1eaea
docs: add note about why we write parquets during transactions
2021-06-16 11:01:14 +02:00
e056d97cf6
test: always test transaction aborts
2021-06-16 11:01:14 +02:00
ec053f674c
feat: make DB catalog work w/ transaction aborts
2021-06-16 11:01:14 +02:00
caaf95c6ec
refactor: remove lock from `TestCatalogState`
2021-06-16 10:51:15 +02:00
c8c412f6fe
refactor: rework catalog state interface
...
This now allows not only for copy-based transaction handling but also
for eager exec and rollbacks. This will be useful to properly implement
transaction aborts for the "real" catalog.
2021-06-16 10:51:15 +02:00
2596de072e
feat: make sure DB catalog can correctly add and remove parquet files
...
Note that this does NOT yet allow it to correctly abort transactions.
2021-06-16 10:50:47 +02:00
bf54ab51f2
refactor: split lifecycle into separate crate ( #1730 )
2021-06-15 15:57:47 +00:00
f96e05d26a
refactor: traitify lifecycle policy ( #1729 )
...
* refactor: traitify lifecycle policy
* chore: docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-15 14:00:06 +00:00
Marco Neumann
Raphael Taylor-Davies
kodiakhq[bot]
Edd Robinson
Marko Mikulicic
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)
Andrew Lamb
Paul Dix
Paul Dix
Jacob Marble