0dffcad109
Merge branch 'main' into ntran/chunkid_catalogchunk
2021-12-03 14:21:04 +00:00
10b1598e68
refactor: remove comments
2021-12-03 09:20:12 -05:00
2857b6a990
Merge branch 'main' into er/feat/load_chunk_cli
2021-12-02 20:20:56 +00:00
b4ea9887ba
refactor: error name
2021-12-02 20:14:02 +00:00
ffc970a60f
feat: add to-be-create-chunk-id for compacting OS chunks in CatalogChunk
2021-12-02 14:49:06 -05:00
1a899b939e
fix: Remove redundant closures identified by clippy
...
https://rust-lang.github.io/rust-clippy/master/index.html#redundant_closure
2021-12-02 11:52:02 -05:00
88aedc556e
feat: add FromStr implementation
2021-12-02 12:59:52 +00:00
e8edab0b43
Merge branch 'main' into ntran/compact_os_preserved_catalog
2021-12-01 18:26:33 +00:00
332485d2c9
fix: use correct `bytes_read` in `DmlMeta`
...
- for file-based write buffers: Use headers + payload
- for Kafka-based write buffers: Use the estimation that we also use for
other metrics
- as a side effect we can now just use `PartialEq` for more types
Fixes #3186 .
2021-12-01 15:57:21 +01:00
6bb7a796e7
chore: Merge branch 'main' into ntran/compact_os_preserved_catalog
2021-11-30 16:55:38 -05:00
e8034cdc53
refactor: handle delete predicates for compacting os
2021-11-30 16:45:04 -05:00
88acf3788e
feat: rebuild catalog ( #3225 ) ( #3253 )
...
* feat: rebuild catalog (#3225 )
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-30 12:17:27 +00:00
23dffefcf8
Merge branch 'main' into crepererum/remove_routing_from_database_mode_7
2021-11-29 11:38:12 +00:00
197634ed50
feat: reload chunk back into read buffer ( #3209 ) ( #3216 )
...
* feat: reload chunk back into read buffer (#3209 )
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
4e043ecb55
refactor: remove old routing / sharding config
...
This is superseded by the new router subsystem.
2021-11-29 12:33:48 +01:00
7f2e4f4342
refactor: remove write buffer direction
...
The direction was required when a database could read or write from/to a
write buffer. Now it is clear from the usage context of a write buffer
context which of the two applications is meant (databases read, routers
write) so the direction flag is no longer required.
2021-11-26 12:38:40 +01:00
a9ec0720b2
feat: allow to format delete predicates as SQL strings
...
This is required to feed them back into the gRPC delete API.
2021-11-23 15:39:53 +01:00
d16a7759ca
Merge branch 'main' into cn/workspace-hack
2021-11-22 17:05:31 +00:00
3b2e5c445b
chore: Merge branch 'main' into ntran/compact_os_framework
2021-11-19 16:18:24 -05:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
e32d367e85
feat: flush delete mailbox on persist ( #3126 ) ( #3147 )
...
* feat: flush delete mailbox on persist (#3126 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-19 09:45:29 +00:00
a5c04e5fe4
feat: framework for compact os chunks
2021-11-17 18:12:51 -05:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
4e71de508e
fix: ensure `ConsistenHasher` is consistent
...
The std `DefaultHasher` is NOT guaranteed to stay the same, so let's
directly use the `SipHasher13` which at the moment (2021-11-15) is used
by the standard lib.
Fixes #3063 .
2021-11-15 17:39:17 +01:00
3d091208af
refactor: move delete predicate into data_types ( #2731 ) ( #3094 )
...
* refactor: move delete predicate into dml (#2731 )
* refactor: move DeletePredicate to data_types
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:28:58 +00:00
da7ba511b2
feat: remove legacy per-line partitioning logic ( #3087 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-11 16:50:42 +00:00
c1fa41196d
feat: impl `Display` for `ShardId`
2021-11-08 14:09:33 +01:00
5b01f9b3a5
docs: explain btree map decision
2021-11-08 11:11:52 +01:00
11f4a1dee8
feat: add connection management for router
2021-11-08 11:11:52 +01:00
06dfdc4af8
feat: simplify sharding ( #3031 )
...
* feat: simplify sharding
* chore: fix lint
2021-11-04 16:57:19 +00:00
4840dc11cb
feat: add basic CRUD operations for router configs
2021-11-04 12:03:14 +01:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
f1a6468e7b
feat: migrate write buffer consumer to use DbWrite ( #2724 ) ( #3003 )
...
* feat: migrate write buffer consumer to use DbWrite (#2724 )
* fix: doc
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-01 16:38:48 +00:00
0bccd35f8c
fix: List details about databases only prints active dbs and their UUIDs
...
We can no longer list deleted databases because the server no longer
knows about them, and we now have UUIDs that are useful to know about,
so change the detailed listing of databases to return UUID instead of
possible deleted at time.
2021-10-28 13:20:29 -04:00
f160712f5e
fix: Remove the concept of generations
...
From iox_object_store and everywhere that fails to compile as a result.
Also make path handling in iox_object_store a bit more consistent.
2021-10-28 13:20:28 -04:00
ec07c11d36
feat: add PartitionWrite ( #2724 ) ( #2974 )
...
* feat: add PartitionWrite (#2724 )
* chore: review feedback
2021-10-28 06:18:10 +00:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
cfd980d1bc
fix: only string columns have distinct counts
2021-10-22 13:04:10 +01:00
d5d885aac6
feat: add MutableBatch::extend_from_range ( #2917 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-21 11:58:11 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
3c953cdf59
feat: improve legibility of task tracker logs ( #2826 )
...
* feat: improve legibility of task tracker logs
* feat: include start time
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-13 14:32:47 +00:00
3dfe400e6b
feat: migrate write path to TimeProvider ( #2722 ) ( #2807 )
2021-10-12 12:09:08 +00:00
b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider ( #2722 ) ( #2798 )
2021-10-11 20:40:00 +00:00
c4a2641764
refactor: remove `time_closed`
...
The "time closed" is a leftover from an old lifecycle system, where
chunks moved through the system (open=>closed=>persisted) without being
merged. Now we have the compaction as well as the split query for
persistence that can merge chunks, so a single "time closed" doesn't
make sense any longer. So in fact it is `None` for many chunks and is
also not persisted. Also the current lifecycle policy doesn't use this
value. So let's just remove it.
Closes #1846 .
2021-10-11 15:49:34 +02:00
f35a49edd0
refactor: move Sequence to data_types ( #2780 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-11 09:23:00 +00:00
64bda1fc08
feat: improve `Debug`/`Display` for test `ChunkId`s
2021-10-08 13:55:56 +02:00
ae7d893199
feat: Add UUID to databases
...
Connects to #2675 .
When a database is created, assign it a UUID and serialize the UUID to
object storage by wrapping the database rules in a new
`PersistedDatabaseRules` type that also contains the UUID.
All APIs to the end user involving rules should continue using only
`DatabaseRules` so the UUID is an internal implementation detail.
2021-10-07 10:19:14 -04:00
27e7a1f925
refactor: Organize use statements
2021-10-07 10:17:19 -04:00
fb9119fa4c
feat: remove serde from data types ( #2652 )
2021-10-07 08:58:06 +00:00
8b06d72a58
fix: address review comments
2021-10-07 10:24:19 +02:00
kodiakhq[bot]
Nga Tran
Edd Robinson
Carol (Nichols || Goulding)
Marco Neumann
Raphael Taylor-Davies
Carol (Nichols || Goulding)