8e69202270
feat: catalog wiping gRPC
2021-06-21 09:31:23 +02:00
d17b5710a8
feat: add server functionality to wipe preserved catalogs
2021-06-21 09:31:23 +02:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
2eb2aca091
fix: fix discrepancy of ckpting config over CLI and protobuf
2021-06-14 10:27:47 +02:00
eae73591f3
feat: add `catalog_checkpoint_interval` lifecycle config
2021-06-14 10:04:50 +02:00
edbaaedfc3
docs: clarify behavior of unspecified transaction encoding
2021-06-10 15:42:21 +02:00
be9b3a4853
fix: protobuf lint fixes
2021-06-10 15:42:21 +02:00
33e364ed78
feat: add encoding info to transaction protobuf
...
This should help with #1381 .
2021-06-10 15:42:21 +02:00
b49abf9b02
Merge branch 'main' into crepererum/lazy_db_loading
2021-06-09 07:23:35 +00:00
4c12fd9b81
fix: Switch tabs for spaces
...
Not sure why my editor did that...
2021-06-07 13:06:50 -04:00
50a69a7f18
fix: Don't mention Kafka unless it's absolutely necessary
2021-06-07 13:01:04 -04:00
2418e91001
feat: Add a DatabaseRule field for an optional Kafka write buffer connection string
2021-06-07 09:56:23 -04:00
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
f38a6378a6
fix: make `getServerStatus` gRPC-compliant
2021-06-04 12:00:49 +02:00
e06d65bb2a
refactor: migrate "DBs initialized" RPC to "server status"
2021-06-04 11:33:41 +02:00
b30d7e2821
feat: move DB loading into background worker
...
Before this change we loaded databases eagerly when a serverID was
passed on startup BEFORE starting up the gRPC server. Since loading
(esp. at its current state without checkpoints and with too many small
parquet files) can take very long, K8s thinks IOx is unhealthy. With
this change we are now loading databases in the server background worker
once a serverID is available. Until then we block all DB-related
interactions including adding new databases (since without inspecting
the object store there is now way we can check if the DB already
exists).
Furthermore we now load database no matter if the serverID was passed on
startup (via CLI or environment variable) or was set later via gRPC
call. Before this change the latter case was somewhat forgotten.
2021-06-04 11:33:41 +02:00
2afa8fa89a
docs: fix typo and mention default
2021-06-03 11:23:29 +02:00
bbd73e59be
feat: jitter background clean-up job + wait on first job
2021-06-03 11:23:29 +02:00
77aeb5ca5d
refactor: use protobuf-native Timestamp instead of string
2021-06-02 09:41:19 +02:00
0a625b50e6
feat: store transaction timestamp in preserved catalog
2021-06-02 09:41:19 +02:00
00e735ef0d
chore: remove unused dependencies ( #1583 )
2021-05-29 10:31:57 +00:00
bae5e5aee3
feat: Add simpler RoutingConfig
2021-05-25 21:51:54 +02:00
9765c53bb4
feat: Implement service readiness API
2021-05-24 17:57:45 +02:00
07db4932ee
refactor: rename data_types/src/chunk.rs -> data_types/src/chunk_metadata.rs ( #1500 )
2021-05-15 10:18:01 +00:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
1a998d4116
feat: preserve parquet metadata in catalog
...
Closes #1380 .
2021-05-07 09:51:44 +02:00
9c9196e8f6
docs: add more info to `catalog.proto`
2021-05-07 09:51:44 +02:00
5db504300d
refactor: use parsed paths instead of raw strings for catalog paths
2021-05-07 09:51:44 +02:00
44de42906f
refactor: use Arc<str> instead of Arc<String> ( #1442 )
2021-05-06 17:05:08 +00:00
216903a949
refactor: move protobuf conversion logic to generated_types ( #1437 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:49:27 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
0d6d94dc00
feat: Enable gRPC reflection
2021-05-05 22:04:47 +02:00
ab142efdd3
feat: add protobuf types for preserved catalog
...
This only includes a minimal type structure to get the transaction layer
going and tested.
See issue #1253 .
2021-05-03 09:51:00 +02:00
979f5f9347
refactor: write buffer to use sequenced entry and new segment
...
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.
Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.
How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
da440d8d27
feat: add a id() method to `management::Operations` ( #1385 )
...
* feat: add Operation::id()
* feat: add a id() method to Operations management client
* fix: broken doc link
2021-04-30 17:09:28 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
262bf446ab
feat: change background worker to use backoff instead of poll ( #1339 ) ( #1347 )
...
* feat: change background worker to use backoff instead (#1339 )
* chore: fix comment
* chore: fix tests
* chore: review comments
* chore: review feedback
2021-04-29 11:10:51 +00:00
1f3bf6b5a6
Merge branch 'main' into sharding_refa
2021-04-28 21:58:21 +00:00
91ee005ada
feat: Add multiple matchers to shard config
2021-04-28 17:02:58 +02:00
b6054f9df9
feat: Make background task period configurable
2021-04-28 15:47:31 +02:00
9b57214344
feat: move closing chunks as soon as possible ( #1329 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
84c863aeb0
refactor: remove unused PersistSegment job ( #1245 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:41:59 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
88ca1a5245
fix: Rename wal.fbs to write_buffer.fbs
2021-04-21 17:43:03 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
ce645262c6
fix: Rename LOCAL env var to INFLUXDB_IOX_INTEGRATION_LOCAL
2021-04-21 14:59:08 +00:00
Marco Neumann
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Marko Mikulicic
Raphael Taylor-Davies
Paul Dix