376471591f
Merge branch 'main' into cn/disown
2021-11-12 19:31:03 +00:00
46c96940f0
feat: Improve log messages to be more specific
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-12 14:21:44 -05:00
63ee5b8317
fix: Shut down databases before deleting or disowning them
2021-11-12 13:37:24 -05:00
56c16414a1
fix: Use shared application time rather than passing in a timestamp
2021-11-12 11:32:02 -05:00
315cc3836b
fix: Combine checks of the database state
2021-11-12 11:17:17 -05:00
8bdfb253ce
fix: Remove context from disown operation
2021-11-12 11:10:20 -05:00
2b8f0b7eeb
fix: Don't hardcode timestamp in restore code
2021-11-11 11:44:19 -05:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
7e13cb77ac
feat: add transactions for context to database owner info on disown
2021-11-10 16:49:32 -05:00
fb7bde527f
feat: Implement disown in Server and Database
2021-11-10 16:49:32 -05:00
489fad0040
feat: Add disown database to the gRPC API
2021-11-10 16:49:31 -05:00
e160954b57
refactor: Extract functions for creating and updating owner info to match the fetching function
2021-11-10 16:49:31 -05:00
d42c890333
refactor: Extract a function for reading a database's owner file
2021-11-10 16:49:31 -05:00
3dda694520
refactor: Extract a macro for matching errors in tests
2021-11-08 11:14:23 -05:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
06dfdc4af8
feat: simplify sharding ( #3031 )
...
* feat: simplify sharding
* chore: fix lint
2021-11-04 16:57:19 +00:00
d28749bd93
refactor: remove Entry from Db interfaces ( #2724 ) ( #3027 )
2021-11-04 13:28:47 +00:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
46213a5a28
chore: update dependencies
2021-11-03 17:00:20 -04:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
6615001dc7
fix: Extract loading the database name out of the rules file into a function
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
207d701bdb
fix: Have DatabaseState keep track of the UUID after reading it from object storage
2021-11-03 15:01:27 -04:00
b1504f17ed
fix: Detangle UUID from ProvidedDatabaseRules
2021-11-03 15:01:27 -04:00
f2cb79bfa0
test: Add more tests around database restore
2021-11-03 15:01:26 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
bbdfc47b02
Merge branch 'main' into crepererum/warn_on_failed_wb_reads
2021-11-03 17:34:14 +00:00
10de63f9b6
refactor: warn when we receive broken data from write buffer
...
We need to understand why this is the case and where broken data gets
stuck, because at the moment it seems that some Kafka hiccups can
poison the write buffer but it seems restarting IOx recovers the
situation.
2021-11-03 16:03:42 +01:00
873ce27b0c
feat: reduce lock metric cardinality ( #3013 )
2021-11-02 17:57:02 +00:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
c724bb2ee8
Merge branch 'main' into dependabot/cargo/tokio-1.13.0
2021-11-01 11:30:33 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
fd5d053d33
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:36:01 +00:00
b1096d2a36
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:26:10 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
6ceab054ab
refactor: move DbWrite to mutable_batch ( #2986 )
...
* refactor: move DbWrite to mutable_batch
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 15:13:05 +00:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
aefdb00bc2
fix: Remove empty test
2021-10-29 09:47:49 -04:00
c02e476175
fix: Split database name from rules at the last possible function
...
When updating database rules, the database name should come from the
rules. Enforce this by only taking the rules as a parameter, rather than
having two arguments that could possibly disagree with each other.
2021-10-29 09:46:23 -04:00
4675dd5331
docs: Add some doc comments ( #2984 )
2021-10-29 11:21:39 +00:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
c61806f919
fix: Apply commenting and logging suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-28 15:32:33 -04:00
f8b5886110
test: Add more test coverage for new behavior
2021-10-28 13:20:29 -04:00
d2bd8f05db
fix: Clean up error messages and comments referencing generations
2021-10-28 13:20:29 -04:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
e80b902d15
test: Delete tests either irrelevant or covered elsewhere
...
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
990f768cda
fix: Assign a UUID when creating a database
2021-10-28 13:20:28 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
cd30e6beab
refactor: use `SpanRecorder` instead of hand-rolled logic
2021-10-25 10:16:05 +02:00
f3cc238d1c
feat: create span contexts on write buffer consumer side
2021-10-25 10:04:08 +02:00
52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate ( #2947 )
...
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate
* fix: make apply_predicate_to_metadata precise
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
f69a470b90
refactor: Use TestDbBuilder to rebuild db in replay tests ( #2950 )
...
* refactor: Use TestDbBuilder to rebuild db in replay tests
* refactor: Organize uses
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:42:05 +00:00
98e84cd624
feat: Only read the server config to list the databases a server owns
2021-10-22 15:56:46 -04:00
9b70001aa4
Merge branch 'main' into cn/remove-list-deleted-databases
2021-10-22 19:36:39 +00:00
948cf92aaa
fix: Remove the ability to list deleted databases
...
Once we're relying on the server config file to know about databases,
IOx shouldn't be listing files in object storage to try and find deleted
databases.
When the disown API is implemented for the floating database design, it
will return the UUID of the database just disowned, so that it can
immediately be used in an adopt API call.
2021-10-22 14:50:06 -04:00
b4506c2f72
docs: Clean up some comments
...
Remove some that weren't accurately describing the code, fix some line
wrapping
2021-10-22 14:34:57 -04:00
739fb075a7
refactor: Organize uses
2021-10-22 14:34:43 -04:00
0c3ace5573
refactor: Call TestDbBuilder.build again instead of making a new builder
2021-10-22 14:32:32 -04:00
e838a22f92
refactor: Make TestDbBuilder reusable for rebuilding databases
...
Many tests want to exercise behavior that happens on reloading or
replay of a database. This changes the way the TestDbBuilder works to
make it possible to call `build` on a builder multiple times to create
databases with the same options to simulate reload of a database without
having to keep track of which pieces need to match.
2021-10-22 14:31:31 -04:00
f4eb3884f3
Merge branch 'main' into cn/write-db-owner-file
2021-10-20 13:51:45 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
10912c2b97
feat: Write owner info in the database's object store directory
...
Use this as an extra check that the database thinks the current server
is its owner when a server tries to initialize and own a database.
If the owner info doesn't match the current server, which could happen
if a server's config was updated but the database's owner info wasn't,
put the database in an error state that requires operator intervention.
2021-10-18 21:13:20 -04:00
c95d6a3a4a
docs: Clarify purpose of new return values
2021-10-18 14:20:18 -04:00
2bb5b4aba4
feat: Read server config file to discover database names and locations on startup
2021-10-18 14:20:18 -04:00
779ba308d8
feat: wire span context frontent=>WriteBuffer
2021-10-18 16:57:00 +02:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
26484309e0
fix: Re-export prost errors instead of wrapping them
2021-10-15 13:44:53 -04:00
2253a7ba62
fix: Use a map in the server config protobuf
2021-10-15 10:52:59 -04:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
5348c9e503
refactor: Move ProstError to root of generated_types to be useful elsewhere
2021-10-15 09:46:20 -04:00
4365dda6cc
test: Remove server init that isn't needed; it gets restarted before anything's checked
2021-10-15 09:46:20 -04:00
42824c30ec
fix: This error is listing databases, not rules
2021-10-15 09:46:20 -04:00
d6b7b56f16
refactor: pull lifecycle out of Db ( #2242 ) ( #2831 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 13:08:00 +00:00
2850487877
feat: make trace collector in Kafka consumer optional
...
The whole application might not have a trace collector configured in
which case we don't wanna produce any spans.
2021-10-15 09:20:40 +02:00
bdd6d67e7a
refactor: split out mutable_batch crate ( #2841 )
...
* refactor: split out mutable_batch crate
* refactor: restore chunk module for better diffs
* chore: fmt
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 16:46:54 +00:00
993c6173d1
Merge branch 'main' into ntran/grpc_storage
2021-10-14 15:28:05 +00:00
faf65f38cc
refactor: address review comments
2021-10-14 11:23:20 -04:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
96e05726ee
refactor: expose negated_predicate API for columns_names
2021-10-14 13:08:56 +01:00
61ec559eee
Merge branch 'main' into crepererum/write_buffer_span_ctx
2021-10-14 11:50:07 +00:00
4087d094b1
refactor: rework write buffer compaction as integration test ( #2830 )
...
* refactor: rework write buffer compaction as integration test
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 11:04:44 +00:00
e911cf9ac1
refactor: make WriteBufferConfigFactory interior mutable ( #2829 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:30:59 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
d752b79cbe
fix: disable persistence during replay ( #2812 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 08:41:10 +00:00
8dd9dcce01
test: verify if all scenarios are created correctly and add a few delete tests for read_filter
2021-10-13 17:21:03 -04:00
a6ca469876
Merge branch 'main' into crepererum/cleanup_pres_catalog_interace
2021-10-13 14:49:02 +00:00
ba829436d7
feat: restart Database ( #2822 ) ( #2825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:23:20 +00:00
1523e0edcd
refactor: clean up preserved catalog interface
...
1. Remove `new_empty` logic. It's a leftover from the time when the
`PreservedCatalog` owned the in-memory catalog.
2. Make `db_name` a part of the `PreservedCatalogConfig`.
2021-10-13 13:58:11 +02:00
d390dfa280
feat: rework delete predicate preservation as integration test ( #2820 )
...
* feat: rework delete predicate preservation as integration test
* chore: review feedback
* chore: fix lint
2021-10-13 10:40:17 +00:00
f7f6965b65
feat: don't panic if `Db::compact_chunks` with no matching chunks ( #2818 )
2021-10-12 21:54:43 +00:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
kodiakhq[bot]
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
Raphael Taylor-Davies
Andrew Lamb
Marco Neumann
dependabot[bot]
Edd Robinson
Andrew Lamb
Carol (Nichols || Goulding)
Nga Tran