757933afc4
fix: use Self when possible
2021-04-19 08:48:11 -04:00
e2954b918e
fix: set defaults for `persist` and `drop_non_persisted` to `true`
...
This should prevent writer deaths for default configs.
Closes #1225 .
2021-04-19 13:03:04 +02:00
8bfcc1782d
feat: Implement sharding logic
2021-04-16 15:14:57 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
4e2d59d9a5
feat: saimplement a few more functions as part of supporting query dfrom parquet files
2021-04-14 16:06:47 -04:00
e516f65cbe
feat: add WriteChunk job
...
For #1083 .
2021-04-14 12:37:43 +02:00
440cef59df
docs: fix docs of `LifecycleRules::buffer_size_soft`
2021-04-14 12:37:43 +02:00
ef7de0aa05
feat: add persist flag to LifecycleRules
...
For #1083 .
2021-04-14 12:37:43 +02:00
bd13c09bad
refactor: make sharder optional when generating entry
2021-04-13 12:52:14 +00:00
7e28f8ef66
feat: Implement Entry writing to Db
...
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.
I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
7f77a01e61
chore: merged main to branch and resolved conflicts
2021-04-12 12:09:04 -04:00
453aeaf1a0
feat: Add tests for writing RB chunks to Object Store
2021-04-09 17:39:23 -04:00
e76980928b
feat: Implement Update API
2021-04-08 22:25:36 +00:00
f501a74aea
refactor: Address review comments
2021-04-07 21:28:03 -04:00
be6e1e48e4
feat: add writer_id and object_store in Db
2021-04-07 18:36:07 -04:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
a7659addbe
chore: cleanup entry.rs based on PR feedback and add better errors
2021-04-06 17:56:58 -04:00
2a134a3ac1
feat: implment line protocol to flatbuffers
...
This implements a builder for converting line protocol to the Entry flatbuffers. It also contains wrapper structs to make working with the flatbuffers a little easier.
The flatbuffer needed an addition to keep track of how many bits in the null mask should be used.
2021-04-06 17:56:56 -04:00
5cd1d6691d
refactor: use DatabaseName in DatabaseRules ( #1127 )
2021-04-06 13:26:30 +00:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
18ca5c5292
refactor: Implement `as_str()` directly rather than using a crate
2021-04-01 21:22:04 +00:00
c0abeba792
feat: system tables
2021-04-01 21:22:04 +00:00
49267114d3
chore: merge main into branch and resolve conflicts
2021-04-01 13:22:49 -04:00
b0e21e5f9e
feat: automated chunk lifecycle ( #1091 )
...
* feat: automated chunk lifecycle
* chore: use >= for lifecycle timestamp comparisons
* chore: review fixes
2021-04-01 11:05:58 +00:00
19a453a483
feat: finally have some framework with clear todos for writing a chunk into parquet files
2021-03-31 16:21:53 -04:00
b61875e0b2
feat: Add TableSummary calculation to ReadBuffer ( #1092 )
2021-03-31 16:26:37 +00:00
7154dfd5f6
feat: Add timestamps to ChunkSummary ( #1079 )
...
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk
* feat: Add timestamps to ChunkSummary
* feat: Add timestamp conversion logic to protobuf types
* test: Add tests
* fix: Update data_types test
* fix: handle negative nanos during conversion
* fix: clippy
* fix: more clippy
* fix: even more clippy
* fix: even more clippy
2021-03-30 19:03:23 +00:00
15e1f9c6ec
refactor: use optional non-zero types ( #1071 )
2021-03-30 11:53:54 +00:00
1a4c793270
feat: lifecycle rules ( #1045 )
2021-03-29 15:32:36 +00:00
98fe75c989
feat: Add ShardConfig to DatabaseRules proto
2021-03-26 14:07:59 +00:00
cda7d13110
refactor: unify serialization of database rules ( #973 ) ( #1047 )
...
* refactor: unify serialization of database rules (#973 )
* chore: fix test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-25 14:13:14 +00:00
8962236e04
fix: Update data_types/src/database_rules.rs
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-22 23:28:00 +00:00
cf51a1a3f1
feat: Add API for ShardConfig
2021-03-22 23:28:00 +00:00
6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types ( #1015 )
...
* refactor: Move some types (not yet exposed to clients) into internal_types
* docs: Add README.md explaining the rationale
* refactor: remove some stragglers
* fix: fix benches
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add clippy lints
* fix: fmt
* docs: Apply suggestions from code review
fix typos
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
dd94a33bc7
feat: retain limited tracker history ( #1005 )
2021-03-17 16:32:34 +00:00
3a53923684
feat: Management API + CLI command to close a chunk and move to read buffer ( #1002 )
...
* feat: Management API + CLI command to close a chunk and move to read buffer
* refactor: Less copy-pasta
* fix: track only once, use `let _` instead of `.ok()`
* docs: Apply suggestions from code review
fix comments ( 🤦♀️ for copy/pasta)
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update server/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Use DatabaseName rather than impl Into<String>
* fix: Fixup logical merge conflicts
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-17 15:25:27 +00:00
3fe1b8c5b7
feat: add longrunning operations client ( #981 )
...
refactor: add separate format feature influxdb_iox_client
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-16 13:19:44 +00:00
65f7a1ac5b
fix: use consistent crate versions ( #989 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 15:42:19 +00:00
facd02a0ec
feat: configure databases created with CLI with reasonable defaults ( #971 )
...
* feat: configure databases created with `database create ...` with reasonable defaults
* fix: remove truncated comments
* fix: Apply suggestions from code review
Improve comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: creates a database without a mutable buffer
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 21:47:43 +00:00
fcd4419702
Merge branch 'main' into pd-routing-rules
2021-03-12 20:02:53 +00:00
2604280476
refactor: simplify to focus on only sharding writes. routes can come later
2021-03-12 14:36:39 -05:00
7e25c4e896
feat: add fanout task tracking ( #956 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-12 15:01:27 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
0606203b40
feat: add configuration for routing rules
...
This is a strawman for what routing rules might look like in DatabaseRules. Once there's a chance for discussion, I'd move next to looking at how the Server would split up an incoming write into separate FB blobs to be sent to remote IOx servers. That might change what the API/configuration looks like as that's how it would be used (at least for writes).
After that it would make sense to move to adding the proto definitions with conversions and gRPC and CLI CRUD to configure routing rules.
2021-03-11 15:25:57 -05:00
65972ccdfc
feat: remove now redundant parts of the HTTP API ( #931 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-10 18:05:39 +00:00
fd4af4b532
Merge branch 'main' into jgm-line-protocol-uinteger
2021-03-09 21:17:40 +00:00
706115178b
chore: remove cruft from config/database rules ( #935 )
...
The replication, query, and subscription concepts here are going to be signficiantly different. Thought it would be best to just remove this cruft for now to avoid confusion.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-09 13:48:59 +00:00
ac1b0c04ae
fix(line-protocol): add unsigned integer field type
...
Fixes #904
The line protocol parser was lacking the unsigned integer type, which
suffixes values with `u`. This adds unsigned integer support to the line
protocol parser, and fills a few corresponding gaps in the mutable
buffer.
2021-03-08 09:59:12 -08:00
Carol (Nichols || Goulding)
Marco Neumann
Marko Mikulicic
Nga Tran
Paul Dix
Raphael Taylor-Davies
Jacob Marble
Andrew Lamb
Raphael Taylor-Davies
Andrew Lamb
kodiakhq[bot]
Paul Dix
Jacob Marble