204dd7c8e9
refactor: Fix some random clippy lints from the future ( #4118 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 09:21:29 +00:00
5585dd3c21
refactor: switch to using DynObjectStore
...
Changes all consumers of the object store to use the dynamically
dispatched DynObjectStore type, instead of using a hardcoded concrete
implementation type.
2022-03-15 16:32:52 +00:00
1d5066c421
refactor: rename ObjectStore -> ObjectStoreImpl
...
Frees up the name for so we can use `dyn ObjectStore` throughout the
code instead of `ObjectStoreApi`.
2022-03-15 16:29:43 +00:00
8f3e44bf76
refactor: Extract a crate for shared data types in the new design
2022-03-02 12:16:15 -05:00
49d1be30e7
feat: wire up `ParquetFilePath` for NG ( #3853 )
...
It's a bit of a duck-type hack, but if we wanna just `ParquetFileChunk`
in the new architecture, we somehow need it to accept new-gen paths.
Also path handling should be somewhat centralized since
ingester/compactor/querier all need to construct them. So having a
`ParquetFilePath` that supports both path styles seems to be a
not-to-bad solution. This should obviously be cleaned up in some
not-to-distant future.
2022-02-24 16:05:38 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
910f381355
refactor: require UUID to create Database ( #3715 )
...
* refactor: require UUID to create Database
* chore: review feedback
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-10 20:04:06 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
0f3cfb024b
test: add database startup recovery test
2021-12-15 12:41:30 +01:00
7499eac067
fix: Disable uuid serde feature; we're not actually serializing any UUIDs
...
Connects to #3117 .
2021-12-06 09:37:31 -05:00
16d8ae5e04
fix: Match tokio features to what's actually in use in each crate
...
Some crates listed features they don't use; other crates ware relying on
feature flags enabled by something else. I tested these changes by
disabling the workspace hack crate and testing each crate.
2021-12-06 09:37:16 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
bca561366b
feat: don't copy parquet files out of disk object store ( #3282 ) ( #3293 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-05 16:31:40 +00:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
25d55cd08a
feat: Move server config paths beneath 'nodes' ( #3144 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:54:32 +00:00
f69d37e9a8
fix: Remove database delete/restore entirely
2021-11-17 12:03:11 -05:00
7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
...
Fixes #2680
2021-11-17 11:41:08 -05:00
3dda694520
refactor: Extract a macro for matching errors in tests
2021-11-08 11:14:23 -05:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
4a225cf913
fix: Change database paths to use UUIDs instead of name
...
This commit changes the iox_object_store crate only; the implications of
these changes are handled in following commits.
Connects to #2675 .
2021-10-28 13:20:28 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
80f04e8fec
docs: Move crate documentation to top of lib.rs
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
062e1ea5b6
fix: Delete now-unused list_possible_databases function
2021-10-25 10:57:40 -04:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
98e84cd624
feat: Only read the server config to list the databases a server owns
2021-10-22 15:56:46 -04:00
948cf92aaa
fix: Remove the ability to list deleted databases
...
Once we're relying on the server config file to know about databases,
IOx shouldn't be listing files in object storage to try and find deleted
databases.
When the disown API is implemented for the floating database design, it
will return the UUID of the database just disowned, so that it can
immediately be used in an adopt API call.
2021-10-22 14:50:06 -04:00
10912c2b97
feat: Write owner info in the database's object store directory
...
Use this as an extra check that the database thinks the current server
is its owner when a server tries to initialize and own a database.
If the owner info doesn't match the current server, which could happen
if a server's config was updated but the database's owner info wasn't,
put the database in an error state that requires operator intervention.
2021-10-18 21:13:20 -04:00
c95d6a3a4a
docs: Clarify purpose of new return values
2021-10-18 14:20:18 -04:00
2bb5b4aba4
feat: Read server config file to discover database names and locations on startup
2021-10-18 14:20:18 -04:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
c64403d8b5
docs: Clarify warning about parsing object storage paths
2021-10-15 11:14:25 -04:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
188fb2282c
feat: Add logic to IoxObjectStore to get and put the server config file
2021-10-15 09:46:20 -04:00
064b7aa613
fix: Remove the database_name method from IoxObjectStore
2021-10-08 15:25:10 -04:00
63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
92583aee82
fix: Remove streaming API since we're not streaming anyway
2021-09-29 08:19:32 -04:00
d05528bcfd
refactor: Use s3_request for put requests
...
Which meant we also needed to change the byte stream to be a closure
that can generate a byte stream
2021-09-29 08:19:32 -04:00
cef5aeee52
refactor: introduce `ChunkId` type
2021-09-20 13:10:41 +02:00
51a40b31bf
feat: Add a --detailed option to the database list CLI
...
That will list both active and deleted databases with their generations.
Closes #2462 .
2021-09-17 15:27:23 -04:00
423a976744
refactor: Use iterator adaptors rather than for loops
2021-09-17 15:27:23 -04:00
44a89cdf75
refactor: Change DeletedDatabase to DetailedDatabase
...
So this info can be reused for active databases in detailed database
lists.
2021-09-17 15:27:22 -04:00
7b6d8f9327
feat: Add an API for restoring a database that was marked deleted
2021-09-15 16:59:37 -04:00
368f0369ee
chore: Rust 1.55
2021-09-10 12:36:49 +02:00
b67610d9b9
chore(deps): bump tokio from 1.10.1 to 1.11.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.10.1 to 1.11.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.10.1...tokio-1.11.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-06 09:11:38 +00:00
ce6030a3cb
feat: Wire list deleted databases through gRPC and CLI APIs
2021-09-02 15:48:07 -04:00
581e49e38f
feat: Add a list_deleted_databases API
...
Fixes #2198 .
2021-09-02 14:57:27 -04:00
Andrew Lamb
Dom Dwyer
Carol (Nichols || Goulding)
Marco Neumann
dependabot[bot]
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
kodiakhq[bot]