07db4932ee
refactor: rename data_types/src/chunk.rs -> data_types/src/chunk_metadata.rs ( #1500 )
2021-05-15 10:18:01 +00:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
1a998d4116
feat: preserve parquet metadata in catalog
...
Closes #1380 .
2021-05-07 09:51:44 +02:00
9c9196e8f6
docs: add more info to `catalog.proto`
2021-05-07 09:51:44 +02:00
5db504300d
refactor: use parsed paths instead of raw strings for catalog paths
2021-05-07 09:51:44 +02:00
44de42906f
refactor: use Arc<str> instead of Arc<String> ( #1442 )
2021-05-06 17:05:08 +00:00
216903a949
refactor: move protobuf conversion logic to generated_types ( #1437 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:49:27 +00:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
0d6d94dc00
feat: Enable gRPC reflection
2021-05-05 22:04:47 +02:00
ab142efdd3
feat: add protobuf types for preserved catalog
...
This only includes a minimal type structure to get the transaction layer
going and tested.
See issue #1253 .
2021-05-03 09:51:00 +02:00
979f5f9347
refactor: write buffer to use sequenced entry and new segment
...
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.
Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.
How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
da440d8d27
feat: add a id() method to `management::Operations` ( #1385 )
...
* feat: add Operation::id()
* feat: add a id() method to Operations management client
* fix: broken doc link
2021-04-30 17:09:28 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
262bf446ab
feat: change background worker to use backoff instead of poll ( #1339 ) ( #1347 )
...
* feat: change background worker to use backoff instead (#1339 )
* chore: fix comment
* chore: fix tests
* chore: review comments
* chore: review feedback
2021-04-29 11:10:51 +00:00
1f3bf6b5a6
Merge branch 'main' into sharding_refa
2021-04-28 21:58:21 +00:00
91ee005ada
feat: Add multiple matchers to shard config
2021-04-28 17:02:58 +02:00
b6054f9df9
feat: Make background task period configurable
2021-04-28 15:47:31 +02:00
9b57214344
feat: move closing chunks as soon as possible ( #1329 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
84c863aeb0
refactor: remove unused PersistSegment job ( #1245 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:41:59 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
88ca1a5245
fix: Rename wal.fbs to write_buffer.fbs
2021-04-21 17:43:03 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
ce645262c6
fix: Rename LOCAL env var to INFLUXDB_IOX_INTEGRATION_LOCAL
2021-04-21 14:59:08 +00:00
d196aacd34
fix: Quote shell variables as recommended by shellcheck
2021-04-21 14:59:08 +00:00
4c58726754
fix: Clean up docker containers in one command instead of two
2021-04-21 14:59:08 +00:00
f3618c54a7
fix: Remove installation of bazel from flatbuffers checking script
...
Because it's now in Docker and working correctly.
2021-04-21 14:59:08 +00:00
9cc5b1950a
fix: Regenerate flatbuffers in a Docker container by default
2021-04-21 14:59:08 +00:00
2934370426
fix: Actually print out the filename flatc is compiling
...
Typed the variable name wrong, oops.
2021-04-21 14:59:08 +00:00
c5d99ba9af
feat: wire up hard buffer limit
...
Fixes #1255 .
2021-04-20 13:25:31 +02:00
e2954b918e
fix: set defaults for `persist` and `drop_non_persisted` to `true`
...
This should prevent writer deaths for default configs.
Closes #1225 .
2021-04-19 13:03:04 +02:00
8bfcc1782d
feat: Implement sharding logic
2021-04-16 15:14:57 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
e516f65cbe
feat: add WriteChunk job
...
For #1083 .
2021-04-14 12:37:43 +02:00
ef7de0aa05
feat: add persist flag to LifecycleRules
...
For #1083 .
2021-04-14 12:37:43 +02:00
59c35e5a4f
feat: Implement WriteBuffer gRPC
2021-04-13 21:43:54 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
eff6eb6b35
docs: Fix instructions on what files are expected to change
2021-04-12 20:16:18 +00:00
37b7bbc069
docs: Improve instructions for usage of the regenerate flatbuffers script
2021-04-12 20:16:18 +00:00
7f77a01e61
chore: merged main to branch and resolved conflicts
2021-04-12 12:09:04 -04:00
453aeaf1a0
feat: Add tests for writing RB chunks to Object Store
2021-04-09 17:39:23 -04:00
d84f05b2f5
chore: Revert "Merge pull request #1159 from influxdata/er/flatbuffers"
...
This reverts commit 0efc1d8174a3e04fecc1
2021-04-09 17:25:59 +00:00
902bdec5e5
fix: git diff-index doesn't actually check the contents but does check timestamps
2021-04-09 14:31:18 +00:00
db639881be
feat: Always rebuild flatbuffers so CI will fail until someone fixes it
2021-04-09 14:31:18 +00:00
1a777a5418
feat: Show more info if flatbuffers check fails
2021-04-09 14:31:18 +00:00
e76980928b
feat: Implement Update API
2021-04-08 22:25:36 +00:00
2bbe5dd46c
chore: update flatc sha
2021-04-08 22:16:41 +01:00
Andrew Lamb
Raphael Taylor-Davies
Marco Neumann
Marko Mikulicic
Paul Dix
kodiakhq[bot]
Carol (Nichols || Goulding)
Nga Tran
Edd Robinson