f4eb3884f3
Merge branch 'main' into cn/write-db-owner-file
2021-10-20 13:51:45 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
10912c2b97
feat: Write owner info in the database's object store directory
...
Use this as an extra check that the database thinks the current server
is its owner when a server tries to initialize and own a database.
If the owner info doesn't match the current server, which could happen
if a server's config was updated but the database's owner info wasn't,
put the database in an error state that requires operator intervention.
2021-10-18 21:13:20 -04:00
c95d6a3a4a
docs: Clarify purpose of new return values
2021-10-18 14:20:18 -04:00
2bb5b4aba4
feat: Read server config file to discover database names and locations on startup
2021-10-18 14:20:18 -04:00
779ba308d8
feat: wire span context frontent=>WriteBuffer
2021-10-18 16:57:00 +02:00
d5ab29711e
fix: Serialize relative db object store paths to the server config
...
So that they can be deserialized, without parsing, to create a new
iox object store from the location listed in the server config.
Notably, the locations serialized don't start with the object storage's
prefix like "s3:" or "file:". The location is the same object storage as
the server configuration that was just read from object storage. Having
the server config on one type of object storage and the database files
on another type is not supported.
2021-10-18 08:37:36 -04:00
26484309e0
fix: Re-export prost errors instead of wrapping them
2021-10-15 13:44:53 -04:00
2253a7ba62
fix: Use a map in the server config protobuf
2021-10-15 10:52:59 -04:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
5348c9e503
refactor: Move ProstError to root of generated_types to be useful elsewhere
2021-10-15 09:46:20 -04:00
4365dda6cc
test: Remove server init that isn't needed; it gets restarted before anything's checked
2021-10-15 09:46:20 -04:00
42824c30ec
fix: This error is listing databases, not rules
2021-10-15 09:46:20 -04:00
d6b7b56f16
refactor: pull lifecycle out of Db ( #2242 ) ( #2831 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 13:08:00 +00:00
2850487877
feat: make trace collector in Kafka consumer optional
...
The whole application might not have a trace collector configured in
which case we don't wanna produce any spans.
2021-10-15 09:20:40 +02:00
bdd6d67e7a
refactor: split out mutable_batch crate ( #2841 )
...
* refactor: split out mutable_batch crate
* refactor: restore chunk module for better diffs
* chore: fmt
* chore: review feedback
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 16:46:54 +00:00
993c6173d1
Merge branch 'main' into ntran/grpc_storage
2021-10-14 15:28:05 +00:00
faf65f38cc
refactor: address review comments
2021-10-14 11:23:20 -04:00
28195b9c0c
chore: new `parquet_catalog` crate
2021-10-14 14:34:59 +02:00
61ec559eee
Merge branch 'main' into crepererum/write_buffer_span_ctx
2021-10-14 11:50:07 +00:00
4087d094b1
refactor: rework write buffer compaction as integration test ( #2830 )
...
* refactor: rework write buffer compaction as integration test
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 11:04:44 +00:00
e911cf9ac1
refactor: make WriteBufferConfigFactory interior mutable ( #2829 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:30:59 +00:00
5e06519afb
feat: propagate trace information through write buffer
2021-10-14 11:07:41 +02:00
d752b79cbe
fix: disable persistence during replay ( #2812 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 08:41:10 +00:00
8dd9dcce01
test: verify if all scenarios are created correctly and add a few delete tests for read_filter
2021-10-13 17:21:03 -04:00
a6ca469876
Merge branch 'main' into crepererum/cleanup_pres_catalog_interace
2021-10-13 14:49:02 +00:00
ba829436d7
feat: restart Database ( #2822 ) ( #2825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:23:20 +00:00
1523e0edcd
refactor: clean up preserved catalog interface
...
1. Remove `new_empty` logic. It's a leftover from the time when the
`PreservedCatalog` owned the in-memory catalog.
2. Make `db_name` a part of the `PreservedCatalogConfig`.
2021-10-13 13:58:11 +02:00
d390dfa280
feat: rework delete predicate preservation as integration test ( #2820 )
...
* feat: rework delete predicate preservation as integration test
* chore: review feedback
* chore: fix lint
2021-10-13 10:40:17 +00:00
f7f6965b65
feat: don't panic if `Db::compact_chunks` with no matching chunks ( #2818 )
2021-10-12 21:54:43 +00:00
8a82f92c5d
refactor: add TimeProvider abstraction ( #2722 ) ( #2815 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 21:19:03 +00:00
5b69bb0d72
feat: reduce lifecycle lock scope ( #2242 ) ( #2810 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 17:34:16 +00:00
8414e6edbb
feat: migrate preserved catalog to TimeProvider ( #2722 ) ( #2808 )
...
* feat: migrate preserved catalog to TimeProvider (#2722 )
* fix: deterministic catalog prune tests
* fix: failing test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-12 14:43:05 +00:00
3dfe400e6b
feat: migrate write path to TimeProvider ( #2722 ) ( #2807 )
2021-10-12 12:09:08 +00:00
0554173684
feat: migrate write buffer to TimeProvider ( #2722 ) ( #2804 )
...
* feat: migrate write buffer to TimeProvider (#2722 )
* chore: review feedback
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2021-10-12 10:32:34 +00:00
b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider ( #2722 ) ( #2798 )
2021-10-11 20:40:00 +00:00
06c2c23322
refactor: create PreservedCatalogConfig struct ( #2793 )
...
* refactor: create PreservedCatalogConfig struct
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 15:43:05 +00:00
05fe4701c1
Merge branch 'main' into dependabot/cargo/cache_loader_async-0.1.2
2021-10-11 15:20:28 +00:00
ad41b74a03
fix: adjust code to `cache_loader_async` 0.1.2
2021-10-11 17:12:08 +02:00
49c63d35b1
chore(deps): bump cache_loader_async from 0.1.1 to 0.1.2
...
Bumps [cache_loader_async](https://github.com/ZeroTwo-Bot/cache-loader-async-rs ) from 0.1.1 to 0.1.2.
- [Release notes](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/releases )
- [Changelog](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ZeroTwo-Bot/cache-loader-async-rs/commits )
---
updated-dependencies:
- dependency-name: cache_loader_async
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-11 15:02:12 +00:00
ae0acf0024
refactor: remove `db_name` param from `select_persistable_chunks`
...
This was only used for logging but is already part of `ChunkAddr`.
2021-10-11 17:01:28 +02:00
8185feddb9
fix: do not break chunk ordering during persistence
...
Fixes #2729 .
2021-10-11 17:01:28 +02:00
5da2f7b1b0
Merge branch 'main' into cn/less-database-name
2021-10-11 10:35:42 -04:00
c4a2641764
refactor: remove `time_closed`
...
The "time closed" is a leftover from an old lifecycle system, where
chunks moved through the system (open=>closed=>persisted) without being
merged. Now we have the compaction as well as the split query for
persistence that can merge chunks, so a single "time closed" doesn't
make sense any longer. So in fact it is `None` for many chunks and is
also not persisted. Also the current lifecycle policy doesn't use this
value. So let's just remove it.
Closes #1846 .
2021-10-11 15:49:34 +02:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
f35a49edd0
refactor: move Sequence to data_types ( #2780 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 09:23:00 +00:00
8407735e00
fix: Pass the database name into PreservedCatalog
2021-10-08 15:25:10 -04:00
276aef69c9
refactor: Move PreservedCatalog test helper functions to test helpers and use them more
2021-10-08 15:25:10 -04:00
f8b5c0ee50
fix: remove obsolete TODO
2021-10-08 12:36:23 +02:00
kodiakhq[bot]
Raphael Taylor-Davies
Andrew Lamb
Carol (Nichols || Goulding)
Marco Neumann
Nga Tran
dependabot[bot]
Carol (Nichols || Goulding)