a2454b542d
fix: Small cleanups in Cargo.tomls ( #3160 )
...
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles
* fix: Alphabetize dependencies
* fix: Add the data_types_conversions feature to get tests passing
* fix: Remove dev dependencies already listed under normal dependencies
* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
c9f02f83e7
Merge branch 'main' into cn/alias-db-commands
2021-11-18 15:13:43 +00:00
ccef3b535a
feat: clean up and add comments for next steps
2021-11-18 10:11:51 -05:00
714fc85c8d
refactor: extract Mailbox type ( #3126 ) ( #3142 )
...
* refactor: extract Mailbox type (#3126 )
* fix: doc
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-18 09:34:06 +00:00
a5c04e5fe4
feat: framework for compact os chunks
2021-11-17 18:12:51 -05:00
f69d37e9a8
fix: Remove database delete/restore entirely
2021-11-17 12:03:11 -05:00
7783e4a7ff
fix: Make delete/restore aliases for release/claim; remove tombstone
...
Fixes #2680
2021-11-17 11:41:08 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
47acd181c5
chore: Update datafusion + arrow/parquet/arrow-flight 6.2.0 ( #3136 )
...
* chore: Update datafusion and arrow
* chore: update arrow/parquet/arrow-flight to 6.2.0
* refactor: Add table_exists to SchemaProvider impl
* fix: clippy
* fix: clippy 2
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-17 14:04:49 +00:00
e6fdd79a0f
feat: emit Kafka stats as metrics instead of logs
...
This maps a subset of Kafka stats as metrics. The set can -- of course
-- be changed in the future depending on our needs.
Fixes #3100 .
2021-11-16 17:18:41 +01:00
553e412226
refactor: DMLOperation write path ( #2731 ) ( #3121 )
...
* refactor: DMLOperation write path (#2731 )
* chore: fmt
* chore: review feedback
2021-11-16 12:42:19 +00:00
bc11244828
feat: Rename database disown/adopt to release/claim ( #3111 )
...
* fix: Rename 'disown' to 'release' database
Connects to #3110
* fix: Rename 'adopt' to 'claim' database
Fixes #3110 .
2021-11-15 20:28:09 +00:00
af28cfa2a6
feat: Add an adopt database API
...
Fixes #2679 .
2021-11-15 09:26:06 -05:00
3d091208af
refactor: move delete predicate into data_types ( #2731 ) ( #3094 )
...
* refactor: move delete predicate into dml (#2731 )
* refactor: move DeletePredicate to data_types
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:28:58 +00:00
a6d83a3026
feat: WriteBufferReader use DmlOperation ( #2731 ) ( #3096 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:19:54 +00:00
48f36fab1f
fix: Box an error so the enum variants don't differ greatly in size
2021-11-12 14:53:07 -05:00
376471591f
Merge branch 'main' into cn/disown
2021-11-12 19:31:03 +00:00
46c96940f0
feat: Improve log messages to be more specific
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-11-12 14:21:44 -05:00
63ee5b8317
fix: Shut down databases before deleting or disowning them
2021-11-12 13:37:24 -05:00
56c16414a1
fix: Use shared application time rather than passing in a timestamp
2021-11-12 11:32:02 -05:00
315cc3836b
fix: Combine checks of the database state
2021-11-12 11:17:17 -05:00
8bdfb253ce
fix: Remove context from disown operation
2021-11-12 11:10:20 -05:00
2b8f0b7eeb
fix: Don't hardcode timestamp in restore code
2021-11-11 11:44:19 -05:00
6f268f8260
refactor: extract DML types ( #2731 ) ( #3084 )
...
* refactor: extract DML types (#2731 )
* chore: fmt
2021-11-11 12:34:07 +00:00
7e13cb77ac
feat: add transactions for context to database owner info on disown
2021-11-10 16:49:32 -05:00
fb7bde527f
feat: Implement disown in Server and Database
2021-11-10 16:49:32 -05:00
489fad0040
feat: Add disown database to the gRPC API
2021-11-10 16:49:31 -05:00
e160954b57
refactor: Extract functions for creating and updating owner info to match the fetching function
2021-11-10 16:49:31 -05:00
d42c890333
refactor: Extract a function for reading a database's owner file
2021-11-10 16:49:31 -05:00
3dda694520
refactor: Extract a macro for matching errors in tests
2021-11-08 11:14:23 -05:00
c8d80e5c28
fix: Change database paths to be under /dbs/ instead of under /[server id]/
2021-11-05 10:14:06 -04:00
898567e221
feat: migrate server to DbWrite ( #2724 ) ( #3035 )
...
* feat: migrate server to DbWrite (#2724 )
* chore: print perf log output
* fix: don't suppress CI status code
* chore: review feedback
* fix: don't error on empty line protocol write payloads
* fix: test
* fix: test
2021-11-05 11:09:33 +00:00
06dfdc4af8
feat: simplify sharding ( #3031 )
...
* feat: simplify sharding
* chore: fix lint
2021-11-04 16:57:19 +00:00
d28749bd93
refactor: remove Entry from Db interfaces ( #2724 ) ( #3027 )
2021-11-04 13:28:47 +00:00
07ba629e2b
feat: migrate write buffer producer to MutableBatch and pbdata ( #2743 ) ( #3021 )
...
* feat: migrate write buffer producer to MutableBatch and pbdata (#2743 )
* fix: Kafka message content type header
* chore: fix doc
2021-11-04 10:20:40 +00:00
46213a5a28
chore: update dependencies
2021-11-03 17:00:20 -04:00
0f62bb1686
fix: Distinguish between restoring twice and name collisions on restore
...
And return them in the correct situations.
2021-11-03 15:02:22 -04:00
6615001dc7
fix: Extract loading the database name out of the rules file into a function
2021-11-03 15:02:21 -04:00
3e44f17698
fix: Return a useful error on attempt to restore an active database
2021-11-03 15:02:18 -04:00
207d701bdb
fix: Have DatabaseState keep track of the UUID after reading it from object storage
2021-11-03 15:01:27 -04:00
b1504f17ed
fix: Detangle UUID from ProvidedDatabaseRules
2021-11-03 15:01:27 -04:00
f2cb79bfa0
test: Add more tests around database restore
2021-11-03 15:01:26 -04:00
62e7376394
fix: Restore database should only need UUID, not database name too
2021-11-03 15:01:26 -04:00
bbdfc47b02
Merge branch 'main' into crepererum/warn_on_failed_wb_reads
2021-11-03 17:34:14 +00:00
10de63f9b6
refactor: warn when we receive broken data from write buffer
...
We need to understand why this is the case and where broken data gets
stuck, because at the moment it seems that some Kafka hiccups can
poison the write buffer but it seems restarting IOx recovers the
situation.
2021-11-03 16:03:42 +01:00
873ce27b0c
feat: reduce lock metric cardinality ( #3013 )
2021-11-02 17:57:02 +00:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
c724bb2ee8
Merge branch 'main' into dependabot/cargo/tokio-1.13.0
2021-11-01 11:30:33 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
fd5d053d33
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:36:01 +00:00
b1096d2a36
Merge branch 'main' into er/feat/read_buffer/meta_delete
2021-11-01 10:26:10 +00:00
a1f978e280
chore(deps): bump tokio-util from 0.6.8 to 0.6.9
...
Bumps [tokio-util](https://github.com/tokio-rs/tokio ) from 0.6.8 to 0.6.9.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9 )
---
updated-dependencies:
- dependency-name: tokio-util
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 01:28:41 +00:00
6ceab054ab
refactor: move DbWrite to mutable_batch ( #2986 )
...
* refactor: move DbWrite to mutable_batch
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 15:13:05 +00:00
fc25e313c6
Merge branch 'main' into cn/iox-obj-sto-refactors
2021-10-29 13:53:26 +00:00
aefdb00bc2
fix: Remove empty test
2021-10-29 09:47:49 -04:00
c02e476175
fix: Split database name from rules at the last possible function
...
When updating database rules, the database name should come from the
rules. Enforce this by only taking the rules as a parameter, rather than
having two arguments that could possibly disagree with each other.
2021-10-29 09:46:23 -04:00
4675dd5331
docs: Add some doc comments ( #2984 )
2021-10-29 11:21:39 +00:00
8a2410e161
feat: mutable batch write entry ( #2724 ) ( #2973 )
...
* feat: mutable batch write entry (#2724 )
* chore: lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-28 20:15:28 +00:00
c61806f919
fix: Apply commenting and logging suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-28 15:32:33 -04:00
f8b5886110
test: Add more test coverage for new behavior
2021-10-28 13:20:29 -04:00
d2bd8f05db
fix: Clean up error messages and comments referencing generations
2021-10-28 13:20:29 -04:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
ce55cf401c
feat: Return UUID when creating a database
...
Seems polite and is useful for some test setup
2021-10-28 13:20:29 -04:00
4077d3a20a
fix: Take a UUID when restoring a database
2021-10-28 13:20:29 -04:00
95ba84edca
fix: Return UUID when a database is deleted
...
So that it can be immediately used to restore elsewhere.
2021-10-28 13:20:29 -04:00
e80b902d15
test: Delete tests either irrelevant or covered elsewhere
...
Now that the database paths use UUIDs instead of names+generation ids,
and that IoxObjectStore checks there are no files in a directory before
it'll create a database directory (and has tests for that).
2021-10-28 13:20:28 -04:00
990f768cda
fix: Assign a UUID when creating a database
2021-10-28 13:20:28 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
8198c1ff2a
refactor: Rename IoxObjectStore constructors to better match what server does with Databases
2021-10-28 13:20:27 -04:00
04b4eeb6ec
Merge branch 'main' into cn/only-read-server
2021-10-25 15:15:17 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
cd30e6beab
refactor: use `SpanRecorder` instead of hand-rolled logic
2021-10-25 10:16:05 +02:00
f3cc238d1c
feat: create span contexts on write buffer consumer side
2021-10-25 10:04:08 +02:00
52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate ( #2947 )
...
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate
* fix: make apply_predicate_to_metadata precise
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
f69a470b90
refactor: Use TestDbBuilder to rebuild db in replay tests ( #2950 )
...
* refactor: Use TestDbBuilder to rebuild db in replay tests
* refactor: Organize uses
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:42:05 +00:00
98e84cd624
feat: Only read the server config to list the databases a server owns
2021-10-22 15:56:46 -04:00
9b70001aa4
Merge branch 'main' into cn/remove-list-deleted-databases
2021-10-22 19:36:39 +00:00
948cf92aaa
fix: Remove the ability to list deleted databases
...
Once we're relying on the server config file to know about databases,
IOx shouldn't be listing files in object storage to try and find deleted
databases.
When the disown API is implemented for the floating database design, it
will return the UUID of the database just disowned, so that it can
immediately be used in an adopt API call.
2021-10-22 14:50:06 -04:00
b4506c2f72
docs: Clean up some comments
...
Remove some that weren't accurately describing the code, fix some line
wrapping
2021-10-22 14:34:57 -04:00
739fb075a7
refactor: Organize uses
2021-10-22 14:34:43 -04:00
0c3ace5573
refactor: Call TestDbBuilder.build again instead of making a new builder
2021-10-22 14:32:32 -04:00
e838a22f92
refactor: Make TestDbBuilder reusable for rebuilding databases
...
Many tests want to exercise behavior that happens on reloading or
replay of a database. This changes the way the TestDbBuilder works to
make it possible to call `build` on a builder multiple times to create
databases with the same options to simulate reload of a database without
having to keep track of which pieces need to match.
2021-10-22 14:31:31 -04:00
f4eb3884f3
Merge branch 'main' into cn/write-db-owner-file
2021-10-20 13:51:45 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
10912c2b97
feat: Write owner info in the database's object store directory
...
Use this as an extra check that the database thinks the current server
is its owner when a server tries to initialize and own a database.
If the owner info doesn't match the current server, which could happen
if a server's config was updated but the database's owner info wasn't,
put the database in an error state that requires operator intervention.
2021-10-18 21:13:20 -04:00
c95d6a3a4a
docs: Clarify purpose of new return values
2021-10-18 14:20:18 -04:00
2bb5b4aba4
feat: Read server config file to discover database names and locations on startup
2021-10-18 14:20:18 -04:00
779ba308d8
feat: wire span context frontent=>WriteBuffer
2021-10-18 16:57:00 +02:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
26484309e0
fix: Re-export prost errors instead of wrapping them
2021-10-15 13:44:53 -04:00
2253a7ba62
fix: Use a map in the server config protobuf
2021-10-15 10:52:59 -04:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
5348c9e503
refactor: Move ProstError to root of generated_types to be useful elsewhere
2021-10-15 09:46:20 -04:00
4365dda6cc
test: Remove server init that isn't needed; it gets restarted before anything's checked
2021-10-15 09:46:20 -04:00
42824c30ec
fix: This error is listing databases, not rules
2021-10-15 09:46:20 -04:00
d6b7b56f16
refactor: pull lifecycle out of Db ( #2242 ) ( #2831 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 13:08:00 +00:00
2850487877
feat: make trace collector in Kafka consumer optional
...
The whole application might not have a trace collector configured in
which case we don't wanna produce any spans.
2021-10-15 09:20:40 +02:00
bdd6d67e7a
refactor: split out mutable_batch crate ( #2841 )
...
* refactor: split out mutable_batch crate
* refactor: restore chunk module for better diffs
* chore: fmt
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 16:46:54 +00:00
993c6173d1
Merge branch 'main' into ntran/grpc_storage
2021-10-14 15:28:05 +00:00
faf65f38cc
refactor: address review comments
2021-10-14 11:23:20 -04:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
96e05726ee
refactor: expose negated_predicate API for columns_names
2021-10-14 13:08:56 +01:00
61ec559eee
Merge branch 'main' into crepererum/write_buffer_span_ctx
2021-10-14 11:50:07 +00:00
4087d094b1
refactor: rework write buffer compaction as integration test ( #2830 )
...
* refactor: rework write buffer compaction as integration test
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 11:04:44 +00:00
e911cf9ac1
refactor: make WriteBufferConfigFactory interior mutable ( #2829 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:30:59 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
d752b79cbe
fix: disable persistence during replay ( #2812 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 08:41:10 +00:00
8dd9dcce01
test: verify if all scenarios are created correctly and add a few delete tests for read_filter
2021-10-13 17:21:03 -04:00
a6ca469876
Merge branch 'main' into crepererum/cleanup_pres_catalog_interace
2021-10-13 14:49:02 +00:00
ba829436d7
feat: restart Database ( #2822 ) ( #2825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:23:20 +00:00
1523e0edcd
refactor: clean up preserved catalog interface
...
1. Remove `new_empty` logic. It's a leftover from the time when the
`PreservedCatalog` owned the in-memory catalog.
2. Make `db_name` a part of the `PreservedCatalogConfig`.
2021-10-13 13:58:11 +02:00
d390dfa280
feat: rework delete predicate preservation as integration test ( #2820 )
...
* feat: rework delete predicate preservation as integration test
* chore: review feedback
* chore: fix lint
2021-10-13 10:40:17 +00:00
f7f6965b65
feat: don't panic if `Db::compact_chunks` with no matching chunks ( #2818 )
2021-10-12 21:54:43 +00:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
5b69bb0d72
feat: reduce lifecycle lock scope ( #2242 ) ( #2810 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 17:34:16 +00:00
8414e6edbb
feat: migrate preserved catalog to TimeProvider ( #2722 ) ( #2808 )
...
* feat: migrate preserved catalog to TimeProvider (#2722 )
* fix: deterministic catalog prune tests
* fix: failing test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 14:43:05 +00:00
3dfe400e6b
feat: migrate write path to TimeProvider ( #2722 ) ( #2807 )
2021-10-12 12:09:08 +00:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider ( #2722 ) ( #2798 )
2021-10-11 20:40:00 +00:00
06c2c23322
refactor: create PreservedCatalogConfig struct ( #2793 )
...
* refactor: create PreservedCatalogConfig struct
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 15:43:05 +00:00
05fe4701c1
Merge branch 'main' into dependabot/cargo/cache_loader_async-0.1.2
2021-10-11 15:20:28 +00:00
ad41b74a03
fix: adjust code to `cache_loader_async` 0.1.2
2021-10-11 17:12:08 +02:00
49c63d35b1
chore(deps): bump cache_loader_async from 0.1.1 to 0.1.2
...
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.1 to 0.1.2.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 15:02:12 +00:00
ae0acf0024
refactor: remove `db_name` param from `select_persistable_chunks`
...
This was only used for logging but is already part of `ChunkAddr`.
2021-10-11 17:01:28 +02:00
8185feddb9
fix: do not break chunk ordering during persistence
...
Fixes #2729 .
2021-10-11 17:01:28 +02:00
5da2f7b1b0
Merge branch 'main' into cn/less-database-name
2021-10-11 10:35:42 -04:00
c4a2641764
refactor: remove `time_closed`
...
The "time closed" is a leftover from an old lifecycle system, where
chunks moved through the system (open=>closed=>persisted) without being
merged. Now we have the compaction as well as the split query for
persistence that can merge chunks, so a single "time closed" doesn't
make sense any longer. So in fact it is `None` for many chunks and is
also not persisted. Also the current lifecycle policy doesn't use this
value. So let's just remove it.
Closes #1846 .
2021-10-11 15:49:34 +02:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
f35a49edd0
refactor: move Sequence to data_types ( #2780 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 09:23:00 +00:00
8407735e00
fix: Pass the database name into PreservedCatalog
2021-10-08 15:25:10 -04:00
276aef69c9
refactor: Move PreservedCatalog test helper functions to test helpers and use them more
2021-10-08 15:25:10 -04:00
f8b5c0ee50
fix: remove obsolete TODO
2021-10-08 12:36:23 +02:00
d3de6bb6e4
refactor: `max_persisted_timestamp` => `flush_timestamp`
...
There might be data left before this timestamp that wasn't persisted
(e.g. incoming data while the persistence was running).
2021-10-08 12:36:23 +02:00
63a932fa37
refactor: "min unpersisted ts" => "max persisted ts"
...
Store the "maximum persisted timestamp" instead of the "minimum
unpersisted timestamp". This avoids the need to calculate the next
timestamp from the current one (which was done via "max TS + 1ns").
The old calculation was prone to overflow panics. Since the
timestamps in this calculation originate from user-provided data (and
not the wall clock), this was an easy DoS vector that could be triggered
via the following line protocol:
```text
table_1 foo=1 <i64::MAX>
```
which is
```text
table_1 foo=1 9223372036854775807
```
Bonus points: the timestamp persisted in the partition
checkpoints is now the very same that was used by the split query during
persistence. Consistence FTW!
Fixes #2225 .
2021-10-08 11:52:49 +02:00
001ed36da4
Merge branch 'main' into crepererum/issue2627
2021-10-08 07:35:08 +00:00
15b396c720
fix: Set a UUID if none was read from object storage on server startup
...
This enables a smoother transition to this state when this gets
deployed.
2021-10-07 10:19:17 -04:00
169f2499d3
fix: Use bytes for UUID in the protobuf instead of string
2021-10-07 10:19:17 -04:00
ae7d893199
feat: Add UUID to databases
...
Connects to #2675 .
When a database is created, assign it a UUID and serialize the UUID to
object storage by wrapping the database rules in a new
`PersistedDatabaseRules` type that also contains the UUID.
All APIs to the end user involving rules should continue using only
`DatabaseRules` so the UUID is an internal implementation detail.
2021-10-07 10:19:14 -04:00
27e7a1f925
refactor: Organize use statements
2021-10-07 10:17:19 -04:00
81c75eec7e
fix: interaction of preservation and delete predicates
...
This is the second part of #2627 .
2021-10-07 11:38:09 +02:00
57b3be3b2d
fix: interaction of compaction and delete predicates
...
- predicates that existed before the compaction can be forgotten since
they are materialized during compaction
- predicates that are added while the compaction is running must be
included into the new chunk
This is the first half of #2627 .
2021-10-07 11:24:58 +02:00
7d6be3f500
Merge branch 'main' into crepererum/issue2748
2021-10-07 09:04:18 +00:00
8b06d72a58
fix: address review comments
2021-10-07 10:24:19 +02:00
63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
0374ba2284
fix: re-enable no longer flaky part of `delete_predicate_preservation`
...
Fix #2748 .
2021-10-07 10:15:49 +02:00
39157828b1
feat: remove remaining usages of Instant ( #2722 ) ( #2749 )
...
* feat: remove remaining usages of Instant (#2722 )
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-06 16:44:02 +00:00
05387f5a70
test: disable running query after re-create db in delete_predicate_preservation to avoid flaky test
2021-10-06 11:17:02 -04:00
ce5b24e65d
refactor: use DateTime<Utc> in PersistenceWindows ( #2722 ) ( #2743 )
...
* refactor: use DateTime<Utc> in PersistenceWindows (#2722 )
* chore: fix benchmark
* chore: fmt
* chore: review feedback
2021-10-06 09:39:32 +00:00
d322069dd4
refactor: move delete predicate persistence into background job
2021-10-06 08:05:38 +02:00
d72a494198
Merge branch 'main' into crepererum/in_mem_expr_part5
2021-10-05 16:20:24 +00:00
d0929e3a34
feat: persist no chunks ( #2712 ) ( #2718 )
...
* feat: persist no chunks (#2712 )
* fix: persist partition
* fix: chunk ordering test
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-05 15:18:35 +00:00
2a584420b3
refactor: make data_types optional dependency ( #2739 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-05 15:07:45 +00:00
bb7a27e5ed
refactor: use proper sets during delete predicate collection
...
We no longer need hacky pointer tricks to de-duplicate delete predicates
when collecting them for catalog checkpoints. This was once required
when the delete predicates didn't implement `Eq` and `Hash` but now it's
all way easier.
2021-10-05 10:37:34 +02:00
28ccf2a8c3
refactor: `TransactionHandle::delete_predicate` cannot fail
2021-10-05 09:41:46 +02:00
10c1a72402
refactor: remove unused fields from `DeletePredicate`
2021-10-05 09:29:24 +02:00
f6fc148fe5
Merge branch 'main' into crepererum/issue2633a
2021-10-04 15:50:48 +00:00
742a1065a1
feat: don't auto-increment background worker now ( #2719 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-04 15:50:09 +00:00
97881079e8
refactor: make `ChunkOrder` non-zero
...
This will make it easier to handle missing values.
Helps with #2633 .
2021-10-04 17:49:12 +02:00
75ac6e8646
refactor: make `DeletePredicate::range` non-optional
2021-10-04 16:36:20 +02:00
5a5a929b9e
refactor: introduce `DeletePredicate`
...
`DeletePredicate` is a simpler version of `Predicate` that is based on
IOx `DeleteExpr` instead of the full-blown DataFusion `Expr`. This will
allow us to do a couple of things (in follow-up changes):
- Order and de-duplicate delete predicates
- Normalize predicates
- Infallible serialization
- Smaller memory footprint
Note that this change only affects delete expressions. Query expressions
that are supported via the API are not changed. The query subsystem also
still uses the full-featured expressions/predicates (delete
expressions/predicates are converted to the more powerful DataFusion
version on-the-fly).
2021-10-04 16:36:20 +02:00
181145eca1
Merge branch 'main' into dependabot/cargo/arrow-5.5.0
2021-10-04 13:10:42 +00:00
7ab10daa19
Merge branch 'main' into dependabot/cargo/arrow-5.5.0
2021-10-04 12:58:29 +01:00
f8c72d611c
Merge branch 'main' into dependabot/cargo/parquet-5.5.0
2021-10-04 12:56:54 +01:00
d0384f60d0
test: update server tests
2021-10-04 12:39:35 +01:00
9cc6b18205
refactor: simplify delete_predicate_preservation test ( #2714 )
2021-10-04 11:38:03 +00:00
e8eab2cc97
feat: allow compaction and persistence to retun no chunk ( #2664 ) ( #2700 )
...
* feat: allow compaction and persistence to retun no chunk (#2664 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-04 10:54:47 +00:00
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
bd22c73b8a
chore: Merge branch 'main' into ntran/delete_endpoint
2021-10-01 13:33:39 -04:00
ee94e9038a
test: finalize codin up delete http endpoints and end-to-end tests
2021-10-01 12:15:00 -04:00
b402423e9e
feat: remove move lifecycle action ( #2674 )
...
* feat: remove move_chunk lifecycle action
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-30 16:58:05 +00:00
003f72ba00
Merge branch 'main' into er/fix/read_buffer/pred_validate
2021-09-29 14:50:12 +01:00
a52b86e070
fix: fallback to no predicate if it can't be validated
...
Closes : #1603
If a predicate cannot be executed against a read buffer chunk because of schema conflicts then fall back to applying no predicate and let the query engine apply predicates in the Filter step of the plan.
2021-09-29 14:42:56 +01:00
92583aee82
fix: Remove streaming API since we're not streaming anyway
2021-09-29 08:19:32 -04:00
d05528bcfd
refactor: Use s3_request for put requests
...
Which meant we also needed to change the byte stream to be a closure
that can generate a byte stream
2021-09-29 08:19:32 -04:00
1534ae9edf
refactor: store chunks in iteration order ( #2634 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-27 11:59:27 +00:00
cf83a325f2
fix: Await on freeze handles instead of try_freeze/returning Transition errors ( #2570 )
...
* fix: Await on a freeze handle instead of returning TransitionInProgress
* fix: Await on freeze handle in skip_replay
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-26 10:44:59 +00:00
621b26166c
feat: validate predicates on read_filter
2021-09-24 14:52:16 +01:00
6d85700e3e
docs: mention return type.
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-09-24 07:23:09 +00:00
4a0cda188e
refactor: make `Partition::force_drop_chunk` similar to `Partition::drop_chunk`
...
- Bubbles up "not found" error, the caller should reason about it
- Returns deleted chunk
2021-09-23 18:37:54 +02:00
e842733c5b
refactor: `CatalogChunk::add_delete_predicate` cannot fail
2021-09-23 09:55:31 +02:00
b16e7ea91a
Merge branch 'main' into crepererum/issue2518c
2021-09-22 16:09:04 +00:00
d38648952c
chore: Update datafusion ( #2602 )
...
* chore: Update datafusion + other deps
* refactor: update query crate for new async interfaces
* refactor: update server crate for new async interface
* refactor: update query_tests crate for new async interfaces
* refactor: update ioxd and server to use new async interface
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-22 10:33:25 +00:00
0d7bb69dd3
feat: teach `Db` to preserve delete predicates
2021-09-22 09:43:37 +02:00
981ee0c6df
refactor: accept unknown chunks in persisted delete predicates
...
Due to the timing of the "persist" lifecycle action and that delete
predicates might arrive at any time + the fact that we don't wanna hold
transaction locks for too long, we should accept delete predicates for
chunks that are currently "persisting" even though that lifecycle action
might fail.
2021-09-22 09:29:50 +02:00
b4b33c378e
test: turn all delete tests on
2021-09-21 15:23:41 -04:00
fb7299a169
fix: bubble up write errors ( #2598 )
...
Fixes #2538 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-21 11:08:37 -04:00
015dfb3b16
test: do not (ab)use the panic hook for replay tests
...
The old construct uses a single assert-statement for both:
- "bubble-up" scenario, were a panic should fire
- a check, were a panic should not fire
That makes it easy to add new tests. However we need two rather
questionable things to make that work:
- catch panic: to convert an assertion to a check
- a custom panic hook: to make tests not overly verbose (aka caught
panics should not show up on stdout)
Esp. the custom panic hook doesn't work too well w/ multi-threaded tests
since it might swallow error messages from unrelated tests and makes
debugging of CI failures hard.
So instead of using assertions for checks, we now implement a proper
assertion and a check for each test. That's a bit more code per check
but easier probably more stable.
2021-09-21 12:00:37 +02:00
77d84ca5ab
Merge branch 'main' into crepererum/chunk_id
2021-09-20 13:39:05 +00:00
c7e6fffaaa
Merge branch 'main' into ntran/delete_scan
2021-09-20 13:29:47 +00:00
cef5aeee52
refactor: introduce `ChunkId` type
2021-09-20 13:10:41 +02:00
140c71eaf0
Merge branch 'main' into crepererum/issue2518a
2021-09-20 09:16:39 +00:00
f62d0eab3c
feat: disable bytes serde ( #2580 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-20 09:07:12 +00:00
acf698c366
fix: delete predicate sorting
2021-09-20 10:48:32 +02:00
492d991f49
feat: delete catalog pres. catalog <=> in-mem catalog API
...
First step towards #2518 . Creates the Rust API to communicate delete
predicates between the preserved catalog and the in-memory catalog and
adds tests ensuring that the in-mem catalog produces the wanted errors
as well as correct checkpoints (similar to how this is done for the
parquet file tracking already).
**This does NOT contain the actual preservation!**
2021-09-20 10:48:32 +02:00
876bb10cf8
chore(deps): bump rand_distr from 0.4.1 to 0.4.2
...
Bumps [rand_distr](https://github.com/rust-random/rand ) from 0.4.1 to 0.4.2.
- [Release notes](https://github.com/rust-random/rand/releases )
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-random/rand/compare/rand_distr-0.4.1...rand_distr-0.4.2 )
---
updated-dependencies:
- dependency-name: rand_distr
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 08:39:39 +00:00
0f5198c88d
test: fix tests dealing w/ parquet metadata sizes
...
Sizes now depend on the actual content and therefore we need
deterministic timestamps.
2021-09-20 09:42:53 +02:00
e15631002e
test: allow test code to specify exact parquet creation timestamp
...
This is required for deterministic sizes since different timestamp lead
to different compression ratios.
2021-09-20 09:42:52 +02:00
364d245eae
feat: apply negated delete predicates during scan
2021-09-17 16:20:42 -04:00
Carol (Nichols || Goulding)
kodiakhq[bot]
Nga Tran
Raphael Taylor-Davies
Carol (Nichols || Goulding)
Andrew Lamb
Marco Neumann
Jake Goulding
Andrew Lamb
dependabot[bot]
Edd Robinson
Carol (Nichols || Goulding)