e5de73133c
feat: Change write buffer connection rule to take either Writing or Reading connection info
...
A database on one IOx server can, exclusively:
- Not interact with Kafka at all
- Send writes to Kafka
- Read writes from Kafka
Notably, a database on a particular server will never write *and* read from Kafka at the same time.
2021-07-08 09:28:34 -04:00
83e50cfba4
refactor: Rename field to not contain the type
2021-07-08 09:28:34 -04:00
7059f16b9e
refactor: Turn mutable_linger_seconds into a non-optional ( #1917 )
2021-07-08 11:25:57 +02:00
e6d995cbd8
chore: Update to Rust 1.53.0 ( #1922 )
...
* chore: Update to Rust 1.53.0
* fix: Update to latest clippy standards
* fix: bad refactor
* fix: Update escaping
* test: update test output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 18:02:03 +00:00
090b0aba11
refactor: remove unused `mutable_size_threshold` lifecycle setting ( #1909 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 17:03:15 +00:00
54fbb60740
feat: expose DB state in gRPC interface
2021-07-02 11:24:36 +02:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
cc038010cd
feat: add persist_age_threshold to LifecycleRules ( #1853 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 21:27:06 +00:00
3ae8ac2467
chore: improve wait_for_chunk failure output ( #1835 )
2021-06-29 11:54:32 +00:00
5287f6a577
feat: print operations on wait_for_chunk failure ( #1809 ) ( #1833 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 11:09:11 +00:00
297fc12db8
feat: compact chunks ( #1776 )
...
* feat: compact chunks
* chore: review feedback
* chore: clippy lints
* chore: document sort key algorithm
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-24 16:49:10 +00:00
c66f9e5aeb
feat: Write entries to Kafka when configured as the write buffer
2021-06-23 10:48:18 -04:00
5cd911c74a
fix: correct row count for object store chunks ( #1789 )
2021-06-23 12:06:49 +00:00
55c546baff
feat: eagerly check object store during CLI `run`
...
Instead of waiting for the server ID to be set and then mark the server
as errored, directly check the object store on startup. This is
important so that we fail fast when Istio isn't up and running yet.
2021-06-22 18:21:30 +02:00
5362c7c924
feat: enable query deduplication ( #1762 )
2021-06-21 18:49:04 +00:00
31ad5c85f9
fix: Consistently refer to docker-compose
2021-06-21 09:41:37 -04:00
b4644e6108
test: Start of Kafka Write Buffer integration tests
2021-06-21 09:41:35 -04:00
a153f841d8
feat: add `--force` flag to CLI wipe command
2021-06-21 09:31:23 +02:00
c0766f1c26
feat: catalog wiping CLI
2021-06-21 09:31:23 +02:00
8e69202270
feat: catalog wiping gRPC
2021-06-21 09:31:23 +02:00
14ba02ec87
feat: expose server and DB init errors over gRPC
...
Closes #1624 .
2021-06-14 17:34:57 +02:00
856751deec
feat: Lifecycle manager unloads, rather than drop, chunks when soft limit is hit ( #1701 )
...
* feat: unload chunks from memory rather than dropping them
* docs: Update server/src/db/lifecycle.rs
Co-authored-by: Marco Neumann <marco@crepererum.net>
* docs: Update comment wording
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-14 13:14:39 +00:00
f4693e36c0
refactor: `catalog_checkpoint_interval` => `catalog_transactions_until_checkpoint`
2021-06-14 10:34:32 +02:00
2eb2aca091
fix: fix discrepancy of ckpting config over CLI and protobuf
2021-06-14 10:27:47 +02:00
4224b693d9
refactor: combine preservation.rs and persistence.rs ( #1692 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-11 11:33:14 +00:00
0cbe74dbde
fix: persistence to parquet by swapping order of arguments ( #1687 )
...
* fix: fix order of arguments
* test: for persistence
2021-06-11 10:55:40 +00:00
11b25b3aaf
refactor: swap order of partition and table in in-memory catalog ( #1678 )
...
* refactor: swap order of partition and table in in-memory catalog
* chore: review feedback
* chore: validate panic message
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-10 16:40:30 +00:00
b49abf9b02
Merge branch 'main' into crepererum/lazy_db_loading
2021-06-09 07:23:35 +00:00
50a69a7f18
fix: Don't mention Kafka unless it's absolutely necessary
2021-06-07 13:01:04 -04:00
2418e91001
feat: Add a DatabaseRule field for an optional Kafka write buffer connection string
2021-06-07 09:56:23 -04:00
87297f7db4
Merge branch 'main' into cn/delete
2021-06-07 13:32:42 +00:00
afe88eeb7c
chore: fix flaky test ( #1643 )
2021-06-07 12:52:11 +00:00
f4a9a5ae56
fix: Remove write buffer
2021-06-04 14:40:17 -04:00
e06d65bb2a
refactor: migrate "DBs initialized" RPC to "server status"
2021-06-04 11:33:41 +02:00
b30d7e2821
feat: move DB loading into background worker
...
Before this change we loaded databases eagerly when a serverID was
passed on startup BEFORE starting up the gRPC server. Since loading
(esp. at its current state without checkpoints and with too many small
parquet files) can take very long, K8s thinks IOx is unhealthy. With
this change we are now loading databases in the server background worker
once a serverID is available. Until then we block all DB-related
interactions including adding new databases (since without inspecting
the object store there is now way we can check if the DB already
exists).
Furthermore we now load database no matter if the serverID was passed on
startup (via CLI or environment variable) or was set later via gRPC
call. Before this change the latter case was somewhat forgotten.
2021-06-04 11:33:41 +02:00
bbd73e59be
feat: jitter background clean-up job + wait on first job
2021-06-03 11:23:29 +02:00
d8f19348bf
feat: per-column dictionaries in MUB ( #1570 )
...
* feat: per-column dictionaries in MUB
* chore: fmt
* refactor: remove chunk-level dictionary
* chore: remove redundant sort
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
bae5e5aee3
feat: Add simpler RoutingConfig
2021-05-25 21:51:54 +02:00
9765c53bb4
feat: Implement service readiness API
2021-05-24 17:57:45 +02:00
5c5064bdac
fix: Set default line timestamp and default partition time to same value ( #1512 )
...
* refactor: Rearrange to allow injection of the current time in tests
* test: Failing test showing a point can be in the wrong partition
* fix: Only get the default time once per ShardedEntry creation, in router
2021-05-24 14:55:11 +00:00
e28c031f76
refactor: make `ServerFixture::restart_server` consume the fixture
...
This also avoids an additional channel lock.
2021-05-20 14:30:30 +02:00
93251f22c7
feat: read perserved catalog during DB startup
...
Closes #1382 .
2021-05-20 10:28:31 +02:00
beee3115f4
feat: expose regex =\~ and to gRPC API
2021-05-07 13:44:51 +01:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
57616c3d16
feat: implement --format and `SET FORMAT <pretty,csv,json>` in SQL repl ( #1404 )
...
* feat: add `set format` and --format flag to sql REPL
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-04 17:02:10 +00:00
f36f1efeeb
feat: add row counts to SQL repl output ( #1405 )
...
* feat: add row counts to repl
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-04 10:57:16 +00:00
3b7c5ac350
fix(storage rpc): do not send back tags with empty values ( #1403 )
2021-05-04 10:35:24 +00:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
da440d8d27
feat: add a id() method to `management::Operations` ( #1385 )
...
* feat: add Operation::id()
* feat: add a id() method to Operations management client
* fix: broken doc link
2021-04-30 17:09:28 +00:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
eb8d91cf1c
refactor: remove additional uses of RecordBatch::try_new ( #1378 )
...
* refactor: remove additional uses of RecordBatch::try_new
* fix: fix accidental change
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 10:24:47 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
c64f25e777
feat: Add `OBSERVER` mode to SQL repl along with SQL primer / cookbook ( #1349 )
...
* feat: add OBSERVER command to repl
* fix: fixup test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 11:47:25 +00:00
1f3bf6b5a6
Merge branch 'main' into sharding_refa
2021-04-28 21:58:21 +00:00
b1924d10b4
fix: Only show current database's system.operations table ( #1310 )
...
* test: add test for operations.system_tables
* fix: only show operations for current database
* fix: update test
* fix: improve test
* refactor: filter in Schema provider rather than in job tracker
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
a64e622f6e
feat: Add interactive SQL repl ( #1332 )
...
* feat: Add interactive SQL repl
* fix: try and fix test
* fix: remove test for prompt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 20:41:02 +00:00
91ee005ada
feat: Add multiple matchers to shard config
2021-04-28 17:02:58 +02:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
74c25f541d
feat: fast MUB dictionary arrow conversion ( #1273 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
83d6550316
feat: Implement write_entry_downstream
2021-04-21 20:50:46 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
c5d99ba9af
feat: wire up hard buffer limit
...
Fixes #1255 .
2021-04-20 13:25:31 +02:00
bada42c0b4
refactor: lower test DB hard limit to 1MB
...
This makes testing the limit faster and helps with #1255 .
2021-04-20 11:43:10 +02:00
15f9c939ef
Merge branch 'main' into refactor-influxdb-client
2021-04-19 17:12:51 +00:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
f092294da3
fix: Use MAX (window end) for timestamps in read group ( #1228 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
4372da2d58
Merge branch 'main' into refactor-influxdb-client
2021-04-16 10:01:42 +05:30
966b97903c
fix: ci tests & write module
...
Add write to api/mod.rs
Rename influx2.rs -> write.rs
Fix end_to_end_cases for DataPoint
Signed-off-by: Aakash Hemadri <aakashhemadri123@gmail.com>
2021-04-16 09:58:23 +05:30
130e059b6b
fix: Flight API should be able to return zero RecordBatches ( #1229 )
2021-04-15 19:10:51 +00:00
c9cbc74857
feat: Use a DID newtype instead of u32 for dictionary ids
...
Rationale
---------
We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
e76980928b
feat: Implement Update API
2021-04-08 22:25:36 +00:00
ef2dc952e1
feat: write once writer id ( #1131 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 17:51:57 +00:00
b0e21e5f9e
feat: automated chunk lifecycle ( #1091 )
...
* feat: automated chunk lifecycle
* chore: use >= for lifecycle timestamp comparisons
* chore: review fixes
2021-04-01 11:05:58 +00:00
7154dfd5f6
feat: Add timestamps to ChunkSummary ( #1079 )
...
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk
* feat: Add timestamps to ChunkSummary
* feat: Add timestamp conversion logic to protobuf types
* test: Add tests
* fix: Update data_types test
* fix: handle negative nanos during conversion
* fix: clippy
* fix: more clippy
* fix: even more clippy
* fix: even more clippy
2021-03-30 19:03:23 +00:00
a630c119ab
feat: make it easy to get OperationMetadata from Operation
2021-03-30 12:57:11 -04:00
569099fc6e
feat: Derive serde for protos
...
Rationale
---------
Our CLI needs to be able to accept configuration as JSON and render configuration as JSON.
Protobufs technically have an official JSON encoding rule called 'jsonpb` but prost doesn't
offer native supprot for it.
`prost` allows us to specify arbitrary derive metadata to be added to generated
code. We emit the `serde` derive directives in the two packages that generate prost code
(`generated_types` and `google_types`).
We use the `serde(rename_all = "camelCase")` to approximate `jsonpb`.
We instruct `prost` to use `bytes::Bytes` for some types, hence we must turn on the `serde` feature
on the `bytes` dependency.
We also use json to serialize the output of the `database get` command, to showcase the feature
and get rid of a TODO. In a subsequent PR I'll teach `database create` (and the yet to be done `database update`) to accept an option JSON configuration body so we can configure partitioning, lifecycle, sharding etc rules etc.
Caveats
-------
This is not technically `jsonpb`. Main issues:
1. default values not omitted
2. no special rendering of special types like `google.protobuf.Any`
Future work
-----------
Figure out if we can get fully compliant `jsonpb`, or at least a decent approximation.
Effect
------
```console
$ cargo run -- database get foobar_weather
{
"name": "foobar_weather",
"partitionTemplate": {
"parts": [
{
"part": {
"time": "%Y-%m-%d %H:00:00"
}
}
]
},
"lifecycleRules": {
"mutableLingerSeconds": 0,
"mutableMinimumAgeSeconds": 0,
"mutableSizeThreshold": 0,
"bufferSizeSoft": 0,
"bufferSizeHard": 0,
"sortOrder": {
"order": 2,
"sort": {
"createdAtTime": {}
}
},
"dropNonPersisted": false,
"immutable": false
},
"walBufferConfig": null,
"shardConfig": {
"specificTargets": null,
"hashRing": null,
"ignoreErrors": false
}
}
```
2021-03-30 15:16:31 +00:00
6a48001d13
refactor: Manage storage directly in the Catalog ( #1057 )
...
* refactor: Manage mutable buffer chunks directly
* fix: do not use mutable_buffer for listing table names
2021-03-29 17:55:07 +00:00
1a4c793270
feat: lifecycle rules ( #1045 )
2021-03-29 15:32:36 +00:00
fb130ea99d
feat: use CatalogProvider and SchemaProvider ( #1058 )
...
* feat: use CatalogProvider and SchemaProvider
* refactor: review comments
2021-03-29 11:08:46 +00:00
98fe75c989
feat: Add ShardConfig to DatabaseRules proto
2021-03-26 14:07:59 +00:00
44d67db733
feat: Rework Db to use Catalog for chunk state ( #1043 )
...
* feat: Rework Db to use Catalog for chunk state
* docs: Update server/src/db.rs
* fix: fmt
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 17:57:11 +00:00
dd94a33bc7
feat: retain limited tracker history ( #1005 )
2021-03-17 16:32:34 +00:00
3a53923684
feat: Management API + CLI command to close a chunk and move to read buffer ( #1002 )
...
* feat: Management API + CLI command to close a chunk and move to read buffer
* refactor: Less copy-pasta
* fix: track only once, use `let _` instead of `.ok()`
* docs: Apply suggestions from code review
fix comments ( 🤦♀️ for copy/pasta)
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update server/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Use DatabaseName rather than impl Into<String>
* fix: Fixup logical merge conflicts
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-17 15:25:27 +00:00
ef9c3f3d8f
feat: Management API + CLI command to list chunks in a partition ( #995 )
...
* feat: Management API + CLI command to list chunks in a partition
For ease of use
* refactor: remove unecessary Result
2021-03-16 20:10:55 +00:00
3fe1b8c5b7
feat: add longrunning operations client ( #981 )
...
refactor: add separate format feature influxdb_iox_client
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-16 13:19:44 +00:00
4781317647
feat: Management API + CLI command to create a new chunk (rollover partition) ( #990 )
...
* feat: Management API + CLI command to create a new chunk (rollover partition)
* fix: Update tests/end_to_end_cases/management_api.rs
fix typo
* fix: logical merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 16:41:18 +00:00
91a92e668a
refactor: complete migration of end-to-end.rs test to use #[tokio::test] ( #987 )
...
* refactor: pull Scenario code out of main module
* refactor: break out http into tests
* refactor: use random org_id and bucket_id
* refactor: port read_api to be indepndent
* refactor: port last test
* refactor: convenience methods to create different clients in end-to-end tests
* fix: improve comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 13:13:55 +00:00
facd02a0ec
feat: configure databases created with CLI with reasonable defaults ( #971 )
...
* feat: configure databases created with `database create ...` with reasonable defaults
* fix: remove truncated comments
* fix: Apply suggestions from code review
Improve comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: creates a database without a mutable buffer
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 21:47:43 +00:00
cc8390f763
refactor: Use prost's enum helper functions for converting from ints
2021-03-12 21:10:10 +00:00
1b36d6b0cd
fix: Return Partition rather than strings
2021-03-12 20:33:39 +00:00
cd66814c3d
feat: Add management API and CLI for listing partitions
2021-03-12 20:33:39 +00:00
6fecf68bd4
feat: make CLI to listing database consistent with other commands ( #974 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-12 14:43:49 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
9df4131e60
feat: Add server remote [set|remove|list] commands
2021-03-12 10:41:18 +00:00
af553f3b38
refactor: run some more end to end tests as tokio tests ( #964 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-11 11:14:49 +00:00
1c6d25e7d8
refactor: Port end-to-end tests that modify global state to their own fixture ( #949 )
...
* refactor: Use autogenerated ports in end-to-end.rs
* refactor: Use different fixture for tests that change writer id
* fix: clippy
2021-03-10 14:05:35 +00:00
Carol (Nichols || Goulding)
Marko Mikulicic
Andrew Lamb
Marco Neumann
Raphael Taylor-Davies
kodiakhq[bot]
Carol (Nichols || Goulding)
Edd Robinson
Aakash Hemadri
Nga Tran
Andrew Lamb