a7659addbe
chore: cleanup entry.rs based on PR feedback and add better errors
2021-04-06 17:56:58 -04:00
2a134a3ac1
feat: implment line protocol to flatbuffers
...
This implements a builder for converting line protocol to the Entry flatbuffers. It also contains wrapper structs to make working with the flatbuffers a little easier.
The flatbuffer needed an addition to keep track of how many bits in the null mask should be used.
2021-04-06 17:56:56 -04:00
5cd1d6691d
refactor: use DatabaseName in DatabaseRules ( #1127 )
2021-04-06 13:26:30 +00:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
18ca5c5292
refactor: Implement `as_str()` directly rather than using a crate
2021-04-01 21:22:04 +00:00
c0abeba792
feat: system tables
2021-04-01 21:22:04 +00:00
49267114d3
chore: merge main into branch and resolve conflicts
2021-04-01 13:22:49 -04:00
b0e21e5f9e
feat: automated chunk lifecycle ( #1091 )
...
* feat: automated chunk lifecycle
* chore: use >= for lifecycle timestamp comparisons
* chore: review fixes
2021-04-01 11:05:58 +00:00
19a453a483
feat: finally have some framework with clear todos for writing a chunk into parquet files
2021-03-31 16:21:53 -04:00
b61875e0b2
feat: Add TableSummary calculation to ReadBuffer ( #1092 )
2021-03-31 16:26:37 +00:00
7154dfd5f6
feat: Add timestamps to ChunkSummary ( #1079 )
...
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk
* feat: Add timestamps to ChunkSummary
* feat: Add timestamp conversion logic to protobuf types
* test: Add tests
* fix: Update data_types test
* fix: handle negative nanos during conversion
* fix: clippy
* fix: more clippy
* fix: even more clippy
* fix: even more clippy
2021-03-30 19:03:23 +00:00
15e1f9c6ec
refactor: use optional non-zero types ( #1071 )
2021-03-30 11:53:54 +00:00
1a4c793270
feat: lifecycle rules ( #1045 )
2021-03-29 15:32:36 +00:00
98fe75c989
feat: Add ShardConfig to DatabaseRules proto
2021-03-26 14:07:59 +00:00
cda7d13110
refactor: unify serialization of database rules ( #973 ) ( #1047 )
...
* refactor: unify serialization of database rules (#973 )
* chore: fix test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-25 14:13:14 +00:00
8962236e04
fix: Update data_types/src/database_rules.rs
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-22 23:28:00 +00:00
cf51a1a3f1
feat: Add API for ShardConfig
2021-03-22 23:28:00 +00:00
6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types ( #1015 )
...
* refactor: Move some types (not yet exposed to clients) into internal_types
* docs: Add README.md explaining the rationale
* refactor: remove some stragglers
* fix: fix benches
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add clippy lints
* fix: fmt
* docs: Apply suggestions from code review
fix typos
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
dd94a33bc7
feat: retain limited tracker history ( #1005 )
2021-03-17 16:32:34 +00:00
3a53923684
feat: Management API + CLI command to close a chunk and move to read buffer ( #1002 )
...
* feat: Management API + CLI command to close a chunk and move to read buffer
* refactor: Less copy-pasta
* fix: track only once, use `let _` instead of `.ok()`
* docs: Apply suggestions from code review
fix comments ( 🤦♀️ for copy/pasta)
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update server/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Use DatabaseName rather than impl Into<String>
* fix: Fixup logical merge conflicts
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-17 15:25:27 +00:00
3fe1b8c5b7
feat: add longrunning operations client ( #981 )
...
refactor: add separate format feature influxdb_iox_client
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-16 13:19:44 +00:00
65f7a1ac5b
fix: use consistent crate versions ( #989 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 15:42:19 +00:00
facd02a0ec
feat: configure databases created with CLI with reasonable defaults ( #971 )
...
* feat: configure databases created with `database create ...` with reasonable defaults
* fix: remove truncated comments
* fix: Apply suggestions from code review
Improve comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: creates a database without a mutable buffer
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 21:47:43 +00:00
fcd4419702
Merge branch 'main' into pd-routing-rules
2021-03-12 20:02:53 +00:00
2604280476
refactor: simplify to focus on only sharding writes. routes can come later
2021-03-12 14:36:39 -05:00
7e25c4e896
feat: add fanout task tracking ( #956 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-12 15:01:27 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
0606203b40
feat: add configuration for routing rules
...
This is a strawman for what routing rules might look like in DatabaseRules. Once there's a chance for discussion, I'd move next to looking at how the Server would split up an incoming write into separate FB blobs to be sent to remote IOx servers. That might change what the API/configuration looks like as that's how it would be used (at least for writes).
After that it would make sense to move to adding the proto definitions with conversions and gRPC and CLI CRUD to configure routing rules.
2021-03-11 15:25:57 -05:00
65972ccdfc
feat: remove now redundant parts of the HTTP API ( #931 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-10 18:05:39 +00:00
fd4af4b532
Merge branch 'main' into jgm-line-protocol-uinteger
2021-03-09 21:17:40 +00:00
706115178b
chore: remove cruft from config/database rules ( #935 )
...
The replication, query, and subscription concepts here are going to be signficiantly different. Thought it would be best to just remove this cruft for now to avoid confusion.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-09 13:48:59 +00:00
ac1b0c04ae
fix(line-protocol): add unsigned integer field type
...
Fixes #904
The line protocol parser was lacking the unsigned integer type, which
suffixes values with `u`. This adds unsigned integer support to the line
protocol parser, and fills a few corresponding gaps in the mutable
buffer.
2021-03-08 09:59:12 -08:00
746373a687
refactor: Remove mutable_buffer crate dependency on query crate ( #927 )
2021-03-05 11:34:27 +00:00
51981c92f5
feat: implement gRPC API and migrate influxdb_iox_client to use it ( #853 )
...
* feat: implement gRPC management API
* feat: migrate influxdb_iox_client to use gRPC API
* fix: review comments
* refactor: separate influxdb_iox_client error types
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-02 17:51:46 +00:00
c7343a4acf
feat: Add iterators over different types of columns in Schema ( #879 )
...
* feat: Add iterators by InfluxColumnType
* fix: Update data_types/src/schema.rs
Implement PR suggestion
Co-authored-by: Edd Robinson <me@edd.io>
* refactor: Remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-26 19:10:28 +00:00
fdcb8baec1
feat: conversion to/from data_types and generated_types ( #848 )
2021-02-26 10:10:47 +00:00
afef7e9ba0
Merge branch 'main' into grpc-management-api-stubs
2021-02-24 14:15:41 +00:00
43371a344c
chore: documentation fixes
2021-02-22 19:38:42 +00:00
0aee605544
Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction
2021-02-22 14:13:34 -05:00
4caa1d732f
refactor: make Selection Copy
2021-02-22 15:32:55 +00:00
d0707725cf
Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction
2021-02-22 10:21:59 -05:00
94a93e56ff
feat: implement `tag_keys` in gRPC planner and across mutable buffer ( #795 )
...
* feat: move tag_column_names into rpc planner
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: compile error
* refactor: remove PassThrough error type
* fix: Avoid extra layers of errors in mutable buffer chunk
* fix: use HashMap::get rather than values() and find
* fix: push filtering down to chunk in gRPC planner
* fix: fixup trait bounds to be non-silly
* fix: remove incorrect comment
* fix: remove cruft
* fix: clippy + fmt
* fix: correct comment
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-15 16:47:52 +00:00
dfe6e50e26
refactor: add arc clone lint
2021-02-15 12:52:12 +00:00
dc465e5d02
feat: Add function to check db size and drop partitions
...
Adds functionality to the server Db to check the mutable buffer size and drop partitions based on the database rules.
2021-02-13 17:19:40 -06:00
83bfa6d949
feat: Add created_at, last_write_at tracking to partition and sorting
...
This commit adds created_at and last_write_at instants to partitions in the mutable buffer. It adds a method on the mutable buffer database to get back the partitions in sorted order based on either the created_at or last_write_at instants. Ordering based on the summary stats from a column are still left to do.
Finally, it modifies the helper function to create replicated write to take a Partitioner trait that can generate partition keys based on lines, rather than taking the DatabaseRules struct directly. This makes it easier to write test cases where data is split into multiple partitions in the mutable buffer.
2021-02-13 17:19:40 -06:00
a03598dfe2
feat: Implement Cross Chunk Schema / RecordBatch merging at query time ( #783 )
...
* feat: feat: Implement Cross Chunk Schema / RecordBatch merging at query time
* docs: update comments about NullArray::new_with-type
* docs: Update comments based on code review
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 18:26:38 +00:00
4c43c311f0
feat: Add database rules configuration for mutable buffer ( #779 )
...
* feat: Add database rules configuration for mutable buffer
* refactor: change all database rules usage to use mutable_buffer_config rather than store_locally
2021-02-10 18:36:29 -05:00
e4baf12c06
feat: Implement database list API in the client.
...
Closes #767
2021-02-10 17:59:19 +01:00
a316b16960
feat: Change table_names to return either Some(set) or None, rather than a plan (try 2) ( #776 )
...
* feat: Change table_names to return either Some(set) or None, rather than a plan
* docs: improve comments
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: merge conflict
* fix: don't clone a string unless needed
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-09 12:20:59 -05:00
Paul Dix
Raphael Taylor-Davies
Jacob Marble
Carol (Nichols || Goulding)
Andrew Lamb
Raphael Taylor-Davies
Nga Tran
Andrew Lamb
Marko Mikulicic
kodiakhq[bot]
Paul Dix
Jacob Marble
Edd Robinson