1156a81567
feat: Add pbjson serialization for storage rpc ( #3324 )
...
* feat: Add pbjson serialization for storage rpc
* chore: update pbjson-build to 0.1.1
Co-authored-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2021-12-07 16:39:16 +00:00
5f3706e0ee
feat: grpc call for copact object store chunks
2021-12-03 18:01:28 -05:00
1e515a1dec
feat: load RUB from object store ( #3224 ) ( #3250 )
2021-11-30 14:39:52 +00:00
23dffefcf8
Merge branch 'main' into crepererum/remove_routing_from_database_mode_7
2021-11-29 11:38:12 +00:00
197634ed50
feat: reload chunk back into read buffer ( #3209 ) ( #3216 )
...
* feat: reload chunk back into read buffer (#3209 )
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-29 11:34:55 +00:00
4e043ecb55
refactor: remove old routing / sharding config
...
This is superseded by the new router subsystem.
2021-11-29 12:33:48 +01:00
1dedfc6697
refactor: remove storage API from pbjson
2021-11-22 22:30:31 +00:00
f328f1bae5
chore: move read source
2021-11-22 21:40:42 +00:00
ddf96efc8a
chore: update storage RPC definitions
2021-11-22 21:40:41 +00:00
a5c04e5fe4
feat: framework for compact os chunks
2021-11-17 18:12:51 -05:00
6320ce6f55
refactor: move delete predicate proto to own package ( #2731 ) ( #3065 )
...
* refactor: move delete predicate proto to own package (#2731 )
* chore: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-08 13:42:26 +00:00
7d070f8582
feat: remove influxdata.iox.write.v1.WriteService ( #3044 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-05 17:51:01 +00:00
511621d3ca
refactor: use stripped-down shard config for router
...
This only implements table-based sharding. We can add value-based
sharding later.
2021-11-02 11:12:20 +01:00
0d0c0cb42b
refactor: move write buffer configs to new home
...
Write buffer configs will partially be shared by database and router
nodes, so lets move them into a shared home.
2021-11-02 10:17:01 +01:00
4c9570b519
refactor: move `catalog` protobuf to `preserved_catalog`
...
This makes it clearer what's going since the contained messages are
only for the preserved part, not the in-mem catalog and its management.
2021-11-01 18:07:25 +01:00
011af2d6ba
feat: add `DeploymentService`
...
Ref #2980 .
2021-11-01 17:55:40 +01:00
8c3446ac87
Merge branch 'main' into crepererum/issue2980b
2021-11-01 16:39:32 +00:00
bcd66c555a
feat: add `RemoteService`
...
Ref #2980 .
2021-11-01 11:38:46 +01:00
83e4514a43
feat: add `DeleteService`
...
Ref #2980 .
2021-10-29 11:57:31 +02:00
7db09316f1
refactor: move router into its own service
2021-10-21 10:35:09 +02:00
c27f096377
feat: rework router config
2021-10-21 10:09:29 +02:00
4f78981958
refactor: factor out partition cfg into its own proto file
2021-10-21 10:09:29 +02:00
1fc00095f5
refactor: factor out write buffer cfg into its own proto file
2021-10-21 10:09:29 +02:00
f75bd0771a
refactor: sort file list in build script
2021-10-21 10:09:29 +02:00
afd6e826e5
feat: Write out server config files listing database name and locations
2021-10-15 09:46:20 -04:00
074ae40382
feat: migrate entry to use bytes::Bytes ( #2842 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-14 10:55:51 +00:00
63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
b8aa4c33ce
refactor: use protobuf bytes for transaction UUIDs
2021-10-05 12:27:48 +02:00
86cee568d5
feat: use upstream pbjson ( #2650 )
...
* feat: use upstream pbjson
* chore: fmt
2021-09-28 16:29:26 +00:00
7e804db0a3
fix: use btree map for some protobuf messages for deterministic outputs
2021-09-20 09:33:18 +02:00
1d55d9a1b5
feat: add pbjson support ( #2468 )
...
* feat: add pbjson support
* chore: fix test
2021-09-16 07:33:27 +00:00
44eb3b994d
feat: `Predicate` serialization
...
Closes #2493 .
2021-09-15 16:37:25 +02:00
27248850e5
refactor: use `byte::Bytes` for metadata in protobuf messages
...
That simplifies printing a bit since we `Vec<u8>` prints quite badly.
2021-09-01 11:26:05 +02:00
98d4c9fca1
feat: switch protobuf write service to canonical definition ( #2182 )
...
* feat: switch protobuf write service to canonical definition
The protobuf definition used for the proto write endpoint was a WIP. Now
that a canonical definition exists at
https://github.com/influxdata/influxdb-pb-data-protocol/ we can switch
to that.
* chore: lint etc
* chore: fix rustdoc nit in proto definition comment
2021-08-04 00:16:49 +00:00
f1a100c6ae
refactor: remove now unused chunk sort order ( #1854 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:39:45 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
4204127b05
refactor: use protobuf for in-parquet metadata
2021-06-30 16:51:37 +02:00
0d6d94dc00
feat: Enable gRPC reflection
2021-05-05 22:04:47 +02:00
ab142efdd3
feat: add protobuf types for preserved catalog
...
This only includes a minimal type structure to get the transaction layer
going and tested.
See issue #1253 .
2021-05-03 09:51:00 +02:00
569099fc6e
feat: Derive serde for protos
...
Rationale
---------
Our CLI needs to be able to accept configuration as JSON and render configuration as JSON.
Protobufs technically have an official JSON encoding rule called 'jsonpb` but prost doesn't
offer native supprot for it.
`prost` allows us to specify arbitrary derive metadata to be added to generated
code. We emit the `serde` derive directives in the two packages that generate prost code
(`generated_types` and `google_types`).
We use the `serde(rename_all = "camelCase")` to approximate `jsonpb`.
We instruct `prost` to use `bytes::Bytes` for some types, hence we must turn on the `serde` feature
on the `bytes` dependency.
We also use json to serialize the output of the `database get` command, to showcase the feature
and get rid of a TODO. In a subsequent PR I'll teach `database create` (and the yet to be done `database update`) to accept an option JSON configuration body so we can configure partitioning, lifecycle, sharding etc rules etc.
Caveats
-------
This is not technically `jsonpb`. Main issues:
1. default values not omitted
2. no special rendering of special types like `google.protobuf.Any`
Future work
-----------
Figure out if we can get fully compliant `jsonpb`, or at least a decent approximation.
Effect
------
```console
$ cargo run -- database get foobar_weather
{
"name": "foobar_weather",
"partitionTemplate": {
"parts": [
{
"part": {
"time": "%Y-%m-%d %H:00:00"
}
}
]
},
"lifecycleRules": {
"mutableLingerSeconds": 0,
"mutableMinimumAgeSeconds": 0,
"mutableSizeThreshold": 0,
"bufferSizeSoft": 0,
"bufferSizeHard": 0,
"sortOrder": {
"order": 2,
"sort": {
"createdAtTime": {}
}
},
"dropNonPersisted": false,
"immutable": false
},
"walBufferConfig": null,
"shardConfig": {
"specificTargets": null,
"hashRing": null,
"ignoreErrors": false
}
}
```
2021-03-30 15:16:31 +00:00
cf51a1a3f1
feat: Add API for ShardConfig
2021-03-22 23:28:00 +00:00
c47a252c15
feat: Check in generated WAL flatbuffers code
2021-03-22 09:38:59 -04:00
cd66814c3d
feat: Add management API and CLI for listing partitions
2021-03-12 20:33:39 +00:00
7e25c4e896
feat: add fanout task tracking ( #956 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-12 15:01:27 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
d2859a99d0
feat: add google longrunning operations stubs ( #959 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-10 17:34:07 +00:00
500c237f62
feat: Add Service + CLI command to write line protocol to IOx ( #939 )
...
* feat: Add Service + CLI command to write line protocol to IOx
* fix: clippy
* fix: protobuf lint
* fix: LinePrtocol typo
2021-03-08 17:35:19 +00:00
ffc20fa821
feat: add basic gRPC health service ( #862 )
...
* feat: add basic gRPC health service
* feat: update README.md add /health to HTTP API
* feat: add health client to influxdb_iox_client
feat: end-to-end test health check service
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 13:24:12 +00:00
1294d28f56
refactor: fix style errors
2021-02-23 10:55:09 +00:00
5aee25451b
feat: gRPC management API definitions
2021-02-22 18:07:36 +00:00
Andrew Lamb
Nga Tran
Raphael Taylor-Davies
kodiakhq[bot]
Marco Neumann
Edd Robinson
Carol (Nichols || Goulding)
Jacob Marble
Marko Mikulicic
Andrew Lamb
Raphael Taylor-Davies