Commit Graph

2378 Commits (64ec45a15423f30577b830a647d7cbaef934f484)

Author SHA1 Message Date
kodiakhq[bot] 64ec45a154
Merge pull request #1174 from influxdata/crepererum/issue1083
feat: RB -> OS lifecycle
2021-04-14 10:42:26 +00:00
Marco Neumann f430013141 refactor: rename `Db::load_chunk_to_object_store` to `Db::write_chunk_to_object_store
For #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann 1c6690ae3b fix: fix constant use in `server::db::tests::read_from_read_buffer` 2021-04-14 12:37:43 +02:00
Marco Neumann 26ce520cf1 feat: hook RB->OS into lifecycle
Fixes #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann 7b0b975433 feat: add Db::load_chunk_to_object_store_in_background
For #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann e516f65cbe feat: add WriteChunk job
For #1083.
2021-04-14 12:37:43 +02:00
Marco Neumann 440cef59df docs: fix docs of `LifecycleRules::buffer_size_soft` 2021-04-14 12:37:43 +02:00
Marco Neumann 84b2ce4571 docs: extend docs of `Db::load_chunk_to_read_buffer_in_background` 2021-04-14 12:37:43 +02:00
Marco Neumann ef7de0aa05 feat: add persist flag to LifecycleRules
For #1083.
2021-04-14 12:37:43 +02:00
Edd Robinson 9834c845db test: add influxrpc tag_values benches
The initial benchmarks look like this on my i9 MBP:

```
Data in one open chunk and one closed chunk of mutable buffer/tag0/no_pred           1.00     91.0±2.55ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag0/with_pred         1.00     11.5±0.72ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/no_pred           1.00    120.3±5.10ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/with_pred         1.00     11.2±0.22ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/no_pred           1.00    203.2±8.45ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/with_pred         1.00     11.2±0.21ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/no_pred      1.00    100.3±3.73ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/with_pred    1.00     31.2±1.80ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/no_pred      1.00    126.7±2.29ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/with_pred    1.00     33.0±1.70ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/no_pred      1.00    212.0±6.86ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/with_pred    1.00     18.1±0.99ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/no_pred                             1.00     98.7±6.08ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/with_pred                           1.00     11.2±0.37ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/no_pred                             1.00    118.9±3.97ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/with_pred                           1.00     11.7±0.64ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/no_pred                             1.00    202.1±8.49ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/with_pred                           1.00     11.1±0.27ms        ? ?/sec
Data in two read buffer chunks/tag0/no_pred                                          1.00    109.2±5.20ms        ? ?/sec
Data in two read buffer chunks/tag0/with_pred                                        1.00     44.2±1.83ms        ? ?/sec
Data in two read buffer chunks/tag1/no_pred                                          1.00    132.9±3.79ms        ? ?/sec
Data in two read buffer chunks/tag1/with_pred                                        1.00     41.7±2.43ms        ? ?/sec
Data in two read buffer chunks/tag2/no_pred                                          1.00    222.4±7.00ms        ? ?/sec
Data in two read buffer chunks/tag2/with_pred                                        1.00     27.9±0.92ms        ? ?/sec
```
2021-04-14 09:36:39 +00:00
Edd Robinson 4db3a4b3b5 test: enable writer to split large batches 2021-04-14 09:36:39 +00:00
Edd Robinson 3cfbfe8ebf refactor: expose query_tests to benches crate 2021-04-14 09:36:39 +00:00
kodiakhq[bot] 8f2cf61e0e
Merge pull request #1201 from influxdata/ntran/query_local_parquet
feat: Add 2 main functions table_schema and table_names for Parquet Chunk to pay a foundation for querying it
2021-04-13 22:42:50 +00:00
kodiakhq[bot] 8e0ee48018
Merge branch 'main' into ntran/query_local_parquet 2021-04-13 22:38:56 +00:00
Marko Mikulicic c9cbc74857 feat: Use a DID newtype instead of u32 for dictionary ids
Rationale
---------

We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
Nga Tran 05bf28ce85 feat: Add 2 main functions table_schema and table_names for Parquet Chunk ato pay a foundation for querying it 2021-04-13 18:23:55 -04:00
Andrew Lamb 150ed4e1d9
refactor: Remove async from `InfluxRPCPlanner` (#1200)
* refactor: Remove async from InfluxRPCPlanner

* fix: make it compile

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 22:17:19 +00:00
Andrew Lamb 6d0467277b
docs: update multi core design doc (#1197)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 21:56:36 +00:00
Marko Mikulicic 59c35e5a4f feat: Implement WriteBuffer gRPC 2021-04-13 21:43:54 +00:00
Raphael Taylor-Davies 7ef490694c
refactor: reduce module coupling in mutable buffer (#1199)
* refactor: reduce module coupling in mutable buffer

* refactor: tweak visibility

* chore: formatting
2021-04-13 20:59:41 +00:00
Raphael Taylor-Davies 61cd745ab6
refactor: remove mutable buffer predicate logic (#1186)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 19:09:36 +00:00
Andrew Lamb 518df742df
chore: update arrow deps (#1195) 2021-04-13 18:05:03 +00:00
Nga Tran c4e56493d8 chore: Merge branch 'ntran/query_local_parquet' of https://github.com/influxdata/influxdb_iox into ntran/query_local_parquet 2021-04-13 13:59:10 -04:00
Nga Tran 4a6d6bd7ad feat: initial work for querying data from parquet file in object store 2021-04-13 13:57:46 -04:00
Nga Tran a68137a661 feat: initial work for querying data from parquet file in object store 2021-04-13 13:47:10 -04:00
Raphael Taylor-Davies 55a77914b1
feat: basic snapshot caching (#1184) 2021-04-13 17:10:28 +00:00
Marko Mikulicic ee46764a2d fix: Use debug level to log requests
Part of #1188
2021-04-13 15:18:46 +00:00
Paul Dix 871de60bb7 chore: remove get_wal_meta test until it gets refactored 2021-04-13 12:52:14 +00:00
Paul Dix bd13c09bad refactor: make sharder optional when generating entry 2021-04-13 12:52:14 +00:00
Paul Dix 4d22982b5f fix: fix broken tests from column refactoring 2021-04-13 12:52:14 +00:00
Paul Dix 7e28f8ef66 feat: Implement Entry writing to Db
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.

I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
Raphael Taylor-Davies 1997324344
feat: mutable buffer snapshotting (#1179)
* feat: mutable buffer snapshotting

* chore: review feedback
2021-04-13 12:14:54 +00:00
Carol (Nichols || Goulding) eff6eb6b35 docs: Fix instructions on what files are expected to change 2021-04-12 20:16:18 +00:00
Carol (Nichols || Goulding) 37b7bbc069 docs: Improve instructions for usage of the regenerate flatbuffers script 2021-04-12 20:16:18 +00:00
Edd Robinson 0f53ba8d9a docs: improve flatc install 2021-04-12 20:16:18 +00:00
kodiakhq[bot] a728494c76
Merge pull request #1180 from influxdata/ntran/fix_flaky_test
fix: check filename to have deterministic output
2021-04-12 19:41:55 +00:00
Nga Tran 494480bc57 fix: check filename to have deterministic output 2021-04-12 15:37:08 -04:00
Paul Dix d67739c873 chore: updates for pr feedback 2021-04-12 18:43:14 +00:00
Paul Dix 5893c17905 refactor: PR feedback and change ClockValue to actual type. 2021-04-12 18:43:14 +00:00
Paul Dix 31115742ec feat: Add writing of Entry structures to MB Chunk
This adds writing of Entry of a vec of TableWriteBatch to the Mutable Buffer Chunk. This is additional to the previous method of writing via ReplicatedWrite. The next step is to remove the old ReplicatedWrite bits.

Test helpers for parsing line protocol into Entry and writing line protocol directly to Chunks have also been added.
2021-04-12 18:43:14 +00:00
Paul Dix 0a3386f24a refactor: Make PartitionWrite, Table, and Column return keys/names 2021-04-12 18:43:14 +00:00
Paul Dix 3f928ed374 refactor: Add ClockValue and WriterId to Entry 2021-04-12 18:43:14 +00:00
Paul Dix dad8d6bafd chore: Update Entry with test helpers 2021-04-12 18:43:14 +00:00
kodiakhq[bot] 64b679ed3f
Merge pull request #1170 from influxdata/ntran/test_write_parquet
feat: Add tests for writing RB chunks to Object Store
2021-04-12 16:13:44 +00:00
Nga Tran 7f77a01e61 chore: merged main to branch and resolved conflicts 2021-04-12 12:09:04 -04:00
Nga Tran e86a02efa7 refactor: Address comments and add more tests 2021-04-12 11:46:31 -04:00
Nga Tran 453aeaf1a0 feat: Add tests for writing RB chunks to Object Store 2021-04-09 17:39:23 -04:00
Carol (Nichols || Goulding) d84f05b2f5 chore: Revert "Merge pull request #1159 from influxdata/er/flatbuffers"
This reverts commit 0efc1d8174, reversing
changes made to a3e04fecc1.
2021-04-09 17:25:59 +00:00
Jake Goulding 16cc37e7f3
chore: update cloud-storage to 0.9 (#1165)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 14:37:51 +00:00
Carol (Nichols || Goulding) 902bdec5e5 fix: git diff-index doesn't actually check the contents but does check timestamps 2021-04-09 14:31:18 +00:00