f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
c724bb2ee8
Merge branch 'main' into dependabot/cargo/tokio-1.13.0
2021-11-01 11:30:33 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
fd5d053d33
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:36:01 +00:00
b1096d2a36
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:26:10 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
6ceab054ab
refactor: move DbWrite to mutable_batch ( #2986 )
...
* refactor: move DbWrite to mutable_batch
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 15:13:05 +00:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
aefdb00bc2
fix: Remove empty test
2021-10-29 09:47:49 -04:00
c02e476175
fix: Split database name from rules at the last possible function
...
When updating database rules, the database name should come from the
rules. Enforce this by only taking the rules as a parameter, rather than
having two arguments that could possibly disagree with each other.
2021-10-29 09:46:23 -04:00
4675dd5331
docs: Add some doc comments ( #2984 )
2021-10-29 11:21:39 +00:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
c61806f919
fix: Apply commenting and logging suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-28 15:32:33 -04:00
f8b5886110
test: Add more test coverage for new behavior
2021-10-28 13:20:29 -04:00
d2bd8f05db
fix: Clean up error messages and comments referencing generations
2021-10-28 13:20:29 -04:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
e80b902d15
test: Delete tests either irrelevant or covered elsewhere
...
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
990f768cda
fix: Assign a UUID when creating a database
2021-10-28 13:20:28 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
cd30e6beab
refactor: use `SpanRecorder` instead of hand-rolled logic
2021-10-25 10:16:05 +02:00
f3cc238d1c
feat: create span contexts on write buffer consumer side
2021-10-25 10:04:08 +02:00
52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate ( #2947 )
...
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate
* fix: make apply_predicate_to_metadata precise
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
f69a470b90
refactor: Use TestDbBuilder to rebuild db in replay tests ( #2950 )
...
* refactor: Use TestDbBuilder to rebuild db in replay tests
* refactor: Organize uses
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:42:05 +00:00
98e84cd624
feat: Only read the server config to list the databases a server owns
2021-10-22 15:56:46 -04:00
9b70001aa4
Merge branch 'main' into cn/remove-list-deleted-databases
2021-10-22 19:36:39 +00:00
948cf92aaa
fix: Remove the ability to list deleted databases
...
Once we're relying on the server config file to know about databases,
IOx shouldn't be listing files in object storage to try and find deleted
databases.
When the disown API is implemented for the floating database design, it
will return the UUID of the database just disowned, so that it can
immediately be used in an adopt API call.
2021-10-22 14:50:06 -04:00
b4506c2f72
docs: Clean up some comments
...
Remove some that weren't accurately describing the code, fix some line
wrapping
2021-10-22 14:34:57 -04:00
739fb075a7
refactor: Organize uses
2021-10-22 14:34:43 -04:00
0c3ace5573
refactor: Call TestDbBuilder.build again instead of making a new builder
2021-10-22 14:32:32 -04:00
e838a22f92
refactor: Make TestDbBuilder reusable for rebuilding databases
...
Many tests want to exercise behavior that happens on reloading or
replay of a database. This changes the way the TestDbBuilder works to
make it possible to call `build` on a builder multiple times to create
databases with the same options to simulate reload of a database without
having to keep track of which pieces need to match.
2021-10-22 14:31:31 -04:00
f4eb3884f3
Merge branch 'main' into cn/write-db-owner-file
2021-10-20 13:51:45 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
10912c2b97
feat: Write owner info in the database's object store directory
...
Use this as an extra check that the database thinks the current server
is its owner when a server tries to initialize and own a database.
If the owner info doesn't match the current server, which could happen
if a server's config was updated but the database's owner info wasn't,
put the database in an error state that requires operator intervention.
2021-10-18 21:13:20 -04:00
c95d6a3a4a
docs: Clarify purpose of new return values
2021-10-18 14:20:18 -04:00
2bb5b4aba4
feat: Read server config file to discover database names and locations on startup
2021-10-18 14:20:18 -04:00
779ba308d8
feat: wire span context frontent=>WriteBuffer
2021-10-18 16:57:00 +02:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
26484309e0
fix: Re-export prost errors instead of wrapping them
2021-10-15 13:44:53 -04:00
2253a7ba62
fix: Use a map in the server config protobuf
2021-10-15 10:52:59 -04:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
5348c9e503
refactor: Move ProstError to root of generated_types to be useful elsewhere
2021-10-15 09:46:20 -04:00
4365dda6cc
test: Remove server init that isn't needed; it gets restarted before anything's checked
2021-10-15 09:46:20 -04:00
42824c30ec
fix: This error is listing databases, not rules
2021-10-15 09:46:20 -04:00
Raphael Taylor-Davies
kodiakhq[bot]
dependabot[bot]
Edd Robinson
Carol (Nichols || Goulding)
Andrew Lamb
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Marco Neumann