0b55cee5f2
feat: skip applying deletes to persisted chunks during replay ( #3125 ) ( #3153 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-22 10:38:25 +00:00
73d60539ad
refactor: use ChunkGenerator in parquet_catalog ( #2209 ) ( #3167 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-22 10:29:33 +00:00
3b2e5c445b
chore: Merge branch 'main' into ntran/compact_os_framework
2021-11-19 16:18:24 -05:00
40ddcf2a2f
refactor: address review comments
2021-11-19 16:08:01 -05:00
ff78f5a4f5
feat: reduce cardinality of job metrics ( #3169 )
...
* feat: reduce cardinality of job metrics
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 19:05:49 +00:00
76d02deb4c
refactor: form `system.column_chunks` column by column rather than row by row ( #3155 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 18:34:29 +00:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
25d55cd08a
feat: Move server config paths beneath 'nodes' ( #3144 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:54:32 +00:00
e32d367e85
feat: flush delete mailbox on persist ( #3126 ) ( #3147 )
...
* feat: flush delete mailbox on persist (#3126 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:45:29 +00:00
c148251dcb
feat: implement step2: compact and persist os chunks
2021-11-18 18:18:55 -05:00
a2454b542d
fix: Small cleanups in Cargo.tomls ( #3160 )
...
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles
* fix: Alphabetize dependencies
* fix: Add the data_types_conversions feature to get tests passing
* fix: Remove dev dependencies already listed under normal dependencies
* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
c9f02f83e7
Merge branch 'main' into cn/alias-db-commands
2021-11-18 15:13:43 +00:00
ccef3b535a
feat: clean up and add comments for next steps
2021-11-18 10:11:51 -05:00
714fc85c8d
refactor: extract Mailbox type ( #3126 ) ( #3142 )
...
* refactor: extract Mailbox type (#3126 )
* fix: doc
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-18 09:34:06 +00:00
a5c04e5fe4
feat: framework for compact os chunks
2021-11-17 18:12:51 -05:00
f69d37e9a8
fix: Remove database delete/restore entirely
2021-11-17 12:03:11 -05:00
7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
...
Fixes #2680
2021-11-17 11:41:08 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
47acd181c5
chore: Update datafusion + arrow/parquet/arrow-flight 6.2.0 ( #3136 )
...
* chore: Update datafusion and arrow
* chore: update arrow/parquet/arrow-flight to 6.2.0
* refactor: Add table_exists to SchemaProvider impl
* fix: clippy
* fix: clippy 2
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-17 14:04:49 +00:00
e6fdd79a0f
feat: emit Kafka stats as metrics instead of logs
...
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.
Fixes #3100 .
2021-11-16 17:18:41 +01:00
553e412226
refactor: DMLOperation write path ( #2731 ) ( #3121 )
...
* refactor: DMLOperation write path (#2731 )
* chore: fmt
* chore: review feedback
2021-11-16 12:42:19 +00:00
bc11244828
feat: Rename database disown/adopt to release/claim ( #3111 )
...
* fix: Rename 'disown' to 'release' database
Connects to #3110
* fix: Rename 'adopt' to 'claim' database
Fixes #3110 .
2021-11-15 20:28:09 +00:00
af28cfa2a6
feat: Add an adopt database API
...
Fixes #2679 .
2021-11-15 09:26:06 -05:00
3d091208af
refactor: move delete predicate into data_types ( #2731 ) ( #3094 )
...
* refactor: move delete predicate into dml (#2731 )
* refactor: move DeletePredicate to data_types
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:28:58 +00:00
a6d83a3026
feat: WriteBufferReader use DmlOperation ( #2731 ) ( #3096 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:19:54 +00:00
48f36fab1f
fix: Box an error so the enum variants don't differ greatly in size
2021-11-12 14:53:07 -05:00
376471591f
Merge branch 'main' into cn/disown
2021-11-12 19:31:03 +00:00
46c96940f0
feat: Improve log messages to be more specific
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-12 14:21:44 -05:00
63ee5b8317
fix: Shut down databases before deleting or disowning them
2021-11-12 13:37:24 -05:00
56c16414a1
fix: Use shared application time rather than passing in a timestamp
2021-11-12 11:32:02 -05:00
315cc3836b
fix: Combine checks of the database state
2021-11-12 11:17:17 -05:00
8bdfb253ce
fix: Remove context from disown operation
2021-11-12 11:10:20 -05:00
2b8f0b7eeb
fix: Don't hardcode timestamp in restore code
2021-11-11 11:44:19 -05:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
7e13cb77ac
feat: add transactions for context to database owner info on disown
2021-11-10 16:49:32 -05:00
fb7bde527f
feat: Implement disown in Server and Database
2021-11-10 16:49:32 -05:00
489fad0040
feat: Add disown database to the gRPC API
2021-11-10 16:49:31 -05:00
e160954b57
refactor: Extract functions for creating and updating owner info to match the fetching function
2021-11-10 16:49:31 -05:00
d42c890333
refactor: Extract a function for reading a database's owner file
2021-11-10 16:49:31 -05:00
3dda694520
refactor: Extract a macro for matching errors in tests
2021-11-08 11:14:23 -05:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
06dfdc4af8
feat: simplify sharding ( #3031 )
...
* feat: simplify sharding
* chore: fix lint
2021-11-04 16:57:19 +00:00
d28749bd93
refactor: remove Entry from Db interfaces ( #2724 ) ( #3027 )
2021-11-04 13:28:47 +00:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
46213a5a28
chore: update dependencies
2021-11-03 17:00:20 -04:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
6615001dc7
fix: Extract loading the database name out of the rules file into a function
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
207d701bdb
fix: Have DatabaseState keep track of the UUID after reading it from object storage
2021-11-03 15:01:27 -04:00
b1504f17ed
fix: Detangle UUID from ProvidedDatabaseRules
2021-11-03 15:01:27 -04:00
f2cb79bfa0
test: Add more tests around database restore
2021-11-03 15:01:26 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
bbdfc47b02
Merge branch 'main' into crepererum/warn_on_failed_wb_reads
2021-11-03 17:34:14 +00:00
10de63f9b6
refactor: warn when we receive broken data from write buffer
...
We need to understand why this is the case and where broken data gets
stuck, because at the moment it seems that some Kafka hiccups can
poison the write buffer but it seems restarting IOx recovers the
situation.
2021-11-03 16:03:42 +01:00
873ce27b0c
feat: reduce lock metric cardinality ( #3013 )
2021-11-02 17:57:02 +00:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
c724bb2ee8
Merge branch 'main' into dependabot/cargo/tokio-1.13.0
2021-11-01 11:30:33 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
fd5d053d33
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:36:01 +00:00
b1096d2a36
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:26:10 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
6ceab054ab
refactor: move DbWrite to mutable_batch ( #2986 )
...
* refactor: move DbWrite to mutable_batch
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 15:13:05 +00:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
aefdb00bc2
fix: Remove empty test
2021-10-29 09:47:49 -04:00
c02e476175
fix: Split database name from rules at the last possible function
...
When updating database rules, the database name should come from the
rules. Enforce this by only taking the rules as a parameter, rather than
having two arguments that could possibly disagree with each other.
2021-10-29 09:46:23 -04:00
4675dd5331
docs: Add some doc comments ( #2984 )
2021-10-29 11:21:39 +00:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
c61806f919
fix: Apply commenting and logging suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-28 15:32:33 -04:00
f8b5886110
test: Add more test coverage for new behavior
2021-10-28 13:20:29 -04:00
d2bd8f05db
fix: Clean up error messages and comments referencing generations
2021-10-28 13:20:29 -04:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
e80b902d15
test: Delete tests either irrelevant or covered elsewhere
...
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
990f768cda
fix: Assign a UUID when creating a database
2021-10-28 13:20:28 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
cd30e6beab
refactor: use `SpanRecorder` instead of hand-rolled logic
2021-10-25 10:16:05 +02:00
f3cc238d1c
feat: create span contexts on write buffer consumer side
2021-10-25 10:04:08 +02:00
52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate ( #2947 )
...
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate
* fix: make apply_predicate_to_metadata precise
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
f69a470b90
refactor: Use TestDbBuilder to rebuild db in replay tests ( #2950 )
...
* refactor: Use TestDbBuilder to rebuild db in replay tests
* refactor: Organize uses
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:42:05 +00:00
98e84cd624
feat: Only read the server config to list the databases a server owns
2021-10-22 15:56:46 -04:00
9b70001aa4
Merge branch 'main' into cn/remove-list-deleted-databases
2021-10-22 19:36:39 +00:00
948cf92aaa
fix: Remove the ability to list deleted databases
...
Once we're relying on the server config file to know about databases,
IOx shouldn't be listing files in object storage to try and find deleted
databases.
When the disown API is implemented for the floating database design, it
will return the UUID of the database just disowned, so that it can
immediately be used in an adopt API call.
2021-10-22 14:50:06 -04:00
b4506c2f72
docs: Clean up some comments
...
Remove some that weren't accurately describing the code, fix some line
wrapping
2021-10-22 14:34:57 -04:00
739fb075a7
refactor: Organize uses
2021-10-22 14:34:43 -04:00
0c3ace5573
refactor: Call TestDbBuilder.build again instead of making a new builder
2021-10-22 14:32:32 -04:00
e838a22f92
refactor: Make TestDbBuilder reusable for rebuilding databases
...
Many tests want to exercise behavior that happens on reloading or
replay of a database. This changes the way the TestDbBuilder works to
make it possible to call `build` on a builder multiple times to create
databases with the same options to simulate reload of a database without
having to keep track of which pieces need to match.
2021-10-22 14:31:31 -04:00
f4eb3884f3
Merge branch 'main' into cn/write-db-owner-file
2021-10-20 13:51:45 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
10912c2b97
feat: Write owner info in the database's object store directory
...
Use this as an extra check that the database thinks the current server
is its owner when a server tries to initialize and own a database.
If the owner info doesn't match the current server, which could happen
if a server's config was updated but the database's owner info wasn't,
put the database in an error state that requires operator intervention.
2021-10-18 21:13:20 -04:00
c95d6a3a4a
docs: Clarify purpose of new return values
2021-10-18 14:20:18 -04:00
2bb5b4aba4
feat: Read server config file to discover database names and locations on startup
2021-10-18 14:20:18 -04:00
779ba308d8
feat: wire span context frontent=>WriteBuffer
2021-10-18 16:57:00 +02:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
26484309e0
fix: Re-export prost errors instead of wrapping them
2021-10-15 13:44:53 -04:00
2253a7ba62
fix: Use a map in the server config protobuf
2021-10-15 10:52:59 -04:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
5348c9e503
refactor: Move ProstError to root of generated_types to be useful elsewhere
2021-10-15 09:46:20 -04:00
4365dda6cc
test: Remove server init that isn't needed; it gets restarted before anything's checked
2021-10-15 09:46:20 -04:00
42824c30ec
fix: This error is listing databases, not rules
2021-10-15 09:46:20 -04:00
d6b7b56f16
refactor: pull lifecycle out of Db ( #2242 ) ( #2831 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 13:08:00 +00:00
2850487877
feat: make trace collector in Kafka consumer optional
...
The whole application might not have a trace collector configured in
which case we don't wanna produce any spans.
2021-10-15 09:20:40 +02:00
bdd6d67e7a
refactor: split out mutable_batch crate ( #2841 )
...
* refactor: split out mutable_batch crate
* refactor: restore chunk module for better diffs
* chore: fmt
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 16:46:54 +00:00
993c6173d1
Merge branch 'main' into ntran/grpc_storage
2021-10-14 15:28:05 +00:00
faf65f38cc
refactor: address review comments
2021-10-14 11:23:20 -04:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
96e05726ee
refactor: expose negated_predicate API for columns_names
2021-10-14 13:08:56 +01:00
61ec559eee
Merge branch 'main' into crepererum/write_buffer_span_ctx
2021-10-14 11:50:07 +00:00
4087d094b1
refactor: rework write buffer compaction as integration test ( #2830 )
...
* refactor: rework write buffer compaction as integration test
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 11:04:44 +00:00
e911cf9ac1
refactor: make WriteBufferConfigFactory interior mutable ( #2829 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:30:59 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
d752b79cbe
fix: disable persistence during replay ( #2812 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 08:41:10 +00:00
8dd9dcce01
test: verify if all scenarios are created correctly and add a few delete tests for read_filter
2021-10-13 17:21:03 -04:00
a6ca469876
Merge branch 'main' into crepererum/cleanup_pres_catalog_interace
2021-10-13 14:49:02 +00:00
ba829436d7
feat: restart Database ( #2822 ) ( #2825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:23:20 +00:00
1523e0edcd
refactor: clean up preserved catalog interface
...
1. Remove `new_empty` logic. It's a leftover from the time when the
`PreservedCatalog` owned the in-memory catalog.
2. Make `db_name` a part of the `PreservedCatalogConfig`.
2021-10-13 13:58:11 +02:00
d390dfa280
feat: rework delete predicate preservation as integration test ( #2820 )
...
* feat: rework delete predicate preservation as integration test
* chore: review feedback
* chore: fix lint
2021-10-13 10:40:17 +00:00
f7f6965b65
feat: don't panic if `Db::compact_chunks` with no matching chunks ( #2818 )
2021-10-12 21:54:43 +00:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
5b69bb0d72
feat: reduce lifecycle lock scope ( #2242 ) ( #2810 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 17:34:16 +00:00
8414e6edbb
feat: migrate preserved catalog to TimeProvider ( #2722 ) ( #2808 )
...
* feat: migrate preserved catalog to TimeProvider (#2722 )
* fix: deterministic catalog prune tests
* fix: failing test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 14:43:05 +00:00
3dfe400e6b
feat: migrate write path to TimeProvider ( #2722 ) ( #2807 )
2021-10-12 12:09:08 +00:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider ( #2722 ) ( #2798 )
2021-10-11 20:40:00 +00:00
06c2c23322
refactor: create PreservedCatalogConfig struct ( #2793 )
...
* refactor: create PreservedCatalogConfig struct
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 15:43:05 +00:00
05fe4701c1
Merge branch 'main' into dependabot/cargo/cache_loader_async-0.1.2
2021-10-11 15:20:28 +00:00
ad41b74a03
fix: adjust code to `cache_loader_async` 0.1.2
2021-10-11 17:12:08 +02:00
49c63d35b1
chore(deps): bump cache_loader_async from 0.1.1 to 0.1.2
...
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.1 to 0.1.2.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 15:02:12 +00:00
ae0acf0024
refactor: remove `db_name` param from `select_persistable_chunks`
...
This was only used for logging but is already part of `ChunkAddr`.
2021-10-11 17:01:28 +02:00
8185feddb9
fix: do not break chunk ordering during persistence
...
Fixes #2729 .
2021-10-11 17:01:28 +02:00
5da2f7b1b0
Merge branch 'main' into cn/less-database-name
2021-10-11 10:35:42 -04:00
c4a2641764
refactor: remove `time_closed`
...
The "time closed" is a leftover from an old lifecycle system, where
chunks moved through the system (open=>closed=>persisted) without being
merged. Now we have the compaction as well as the split query for
persistence that can merge chunks, so a single "time closed" doesn't
make sense any longer. So in fact it is `None` for many chunks and is
also not persisted. Also the current lifecycle policy doesn't use this
value. So let's just remove it.
Closes #1846 .
2021-10-11 15:49:34 +02:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
f35a49edd0
refactor: move Sequence to data_types ( #2780 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 09:23:00 +00:00
8407735e00
fix: Pass the database name into PreservedCatalog
2021-10-08 15:25:10 -04:00
276aef69c9
refactor: Move PreservedCatalog test helper functions to test helpers and use them more
2021-10-08 15:25:10 -04:00
f8b5c0ee50
fix: remove obsolete TODO
2021-10-08 12:36:23 +02:00
d3de6bb6e4
refactor: `max_persisted_timestamp` => `flush_timestamp`
...
There might be data left before this timestamp that wasn't persisted
(e.g. incoming data while the persistence was running).
2021-10-08 12:36:23 +02:00
63a932fa37
refactor: "min unpersisted ts" => "max persisted ts"
...
Store the "maximum persisted timestamp" instead of the "minimum
unpersisted timestamp". This avoids the need to calculate the next
timestamp from the current one (which was done via "max TS + 1ns").
The old calculation was prone to overflow panics. Since the
timestamps in this calculation originate from user-provided data (and
not the wall clock), this was an easy DoS vector that could be triggered
via the following line protocol:
```text
table_1 foo=1 <i64::MAX>
```
which is
```text
table_1 foo=1 9223372036854775807
```
Bonus points: the timestamp persisted in the partition
checkpoints is now the very same that was used by the split query during
persistence. Consistence FTW!
Fixes #2225 .
2021-10-08 11:52:49 +02:00
001ed36da4
Merge branch 'main' into crepererum/issue2627
2021-10-08 07:35:08 +00:00
15b396c720
fix: Set a UUID if none was read from object storage on server startup
...
This enables a smoother transition to this state when this gets
deployed.
2021-10-07 10:19:17 -04:00
169f2499d3
fix: Use bytes for UUID in the protobuf instead of string
2021-10-07 10:19:17 -04:00
ae7d893199
feat: Add UUID to databases
...
Connects to #2675 .
When a database is created, assign it a UUID and serialize the UUID to
object storage by wrapping the database rules in a new
`PersistedDatabaseRules` type that also contains the UUID.
All APIs to the end user involving rules should continue using only
`DatabaseRules` so the UUID is an internal implementation detail.
2021-10-07 10:19:14 -04:00
Raphael Taylor-Davies
Nga Tran
Andrew Lamb
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
kodiakhq[bot]
Marco Neumann
Jake Goulding
Andrew Lamb
dependabot[bot]
Edd Robinson
Carol (Nichols || Goulding)