Commit Graph

221 Commits (91df8a30e7f0a5fc258f3260e31f4fcea77abd30)

Author SHA1 Message Date
Marco Neumann bbd73e59be feat: jitter background clean-up job + wait on first job 2021-06-03 11:23:29 +02:00
Raphael Taylor-Davies d8f19348bf
feat: per-column dictionaries in MUB (#1570)
* feat: per-column dictionaries in MUB

* chore: fmt

* refactor: remove chunk-level dictionary

* chore: remove redundant sort

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-28 13:51:56 +00:00
Marko Mikulicic bae5e5aee3
feat: Add simpler RoutingConfig 2021-05-25 21:51:54 +02:00
Marko Mikulicic 9765c53bb4
feat: Implement service readiness API 2021-05-24 17:57:45 +02:00
Carol (Nichols || Goulding) 5c5064bdac
fix: Set default line timestamp and default partition time to same value (#1512)
* refactor: Rearrange to allow injection of the current time in tests

* test: Failing test showing a point can be in the wrong partition

* fix: Only get the default time once per ShardedEntry creation, in router
2021-05-24 14:55:11 +00:00
Marco Neumann e28c031f76 refactor: make `ServerFixture::restart_server` consume the fixture
This also avoids an additional channel lock.
2021-05-20 14:30:30 +02:00
Marco Neumann 93251f22c7 feat: read perserved catalog during DB startup
Closes #1382.
2021-05-20 10:28:31 +02:00
Marco Neumann 25ec0ab4ca test: make end2end server fixture restartable 2021-05-20 10:22:20 +02:00
Edd Robinson beee3115f4 feat: expose regex =\~ and to gRPC API 2021-05-07 13:44:51 +01:00
Raphael Taylor-Davies 9320f59de0
feat: add shard sink indirection (#1447)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
Raphael Taylor-Davies 10f89a3e8d
refactor: split entry out into separate crate (#1428)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
Raphael Taylor-Davies 411cf134e9
refactor: explode arrow_deps (#1425)
* refactor: explode arrow_deps

* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
Andrew Lamb 57616c3d16
feat: implement --format and `SET FORMAT <pretty,csv,json>` in SQL repl (#1404)
* feat: add `set format` and --format flag to sql REPL

* fix: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-04 17:02:10 +00:00
Andrew Lamb f36f1efeeb
feat: add row counts to SQL repl output (#1405)
* feat: add row counts to repl

* fix: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-04 10:57:16 +00:00
Andrew Lamb 3b7c5ac350
fix(storage rpc): do not send back tags with empty values (#1403) 2021-05-04 10:35:24 +00:00
Raphael Taylor-Davies a967ebfabd
refactor: rename closing to closed (#1396)
* refactor: rename closing to closed

* refactor: further renames
2021-04-30 20:59:45 +00:00
Andrew Lamb da440d8d27
feat: add a id() method to `management::Operations` (#1385)
* feat: add Operation::id()

* feat: add a id() method to Operations management client

* fix: broken doc link
2021-04-30 17:09:28 +00:00
Andrew Lamb 40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq (#1368) 2021-04-30 10:51:08 +00:00
Andrew Lamb eb8d91cf1c
refactor: remove additional uses of RecordBatch::try_new (#1378)
* refactor: remove additional uses of RecordBatch::try_new

* fix: fix accidental change

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 10:24:47 +00:00
Andrew Lamb 0788892413
feat: add row_count to system.chunks and Chunk managment API (#1373)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
Andrew Lamb c64f25e777
feat: Add `OBSERVER` mode to SQL repl along with SQL primer / cookbook (#1349)
* feat: add OBSERVER command to repl

* fix: fixup test

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 11:47:25 +00:00
kodiakhq[bot] 1f3bf6b5a6
Merge branch 'main' into sharding_refa 2021-04-28 21:58:21 +00:00
Andrew Lamb b1924d10b4
fix: Only show current database's system.operations table (#1310)
* test: add test for operations.system_tables

* fix: only show operations for current database

* fix: update test

* fix: improve test

* refactor: filter in Schema provider rather than in job tracker

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
Andrew Lamb a64e622f6e
feat: Add interactive SQL repl (#1332)
* feat: Add interactive SQL repl

* fix: try and fix test

* fix: remove test for prompt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 20:41:02 +00:00
Edd Robinson 5dfe312867 test: add benchmark for influxrpc read_filter 2021-04-28 16:11:57 +00:00
Marko Mikulicic 91ee005ada
feat: Add multiple matchers to shard config 2021-04-28 17:02:58 +02:00
Carol (Nichols || Goulding) 272cdb85ce fix: Use the ServerId type everywhere, for writing, querying, anything 2021-04-26 18:44:32 +00:00
Raphael Taylor-Davies 0a835436ac
feat: use bitmasks within MUB (#1274) (#1289)
* feat: use bitmasks within MUB (#1274)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
Marco Neumann f12f1eeef7 refactor: make chunks per-table
This changes the hierarchy from

```
database -> partition -> chunk -> table
```

to

```
database -> partition -> table -> chunk
```

Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).

Closes #1256.
2021-04-23 09:18:55 +02:00
Raphael Taylor-Davies 74c25f541d
feat: fast MUB dictionary arrow conversion (#1273)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
Marko Mikulicic 83d6550316 feat: Implement write_entry_downstream 2021-04-21 20:50:46 +00:00
Carol (Nichols || Goulding) ee45db402b refactor: Rename WalBufferConfig to WriteBufferConfig 2021-04-21 17:43:03 +00:00
kodiakhq[bot] dc6637b448
Merge branch 'main' into jgm-tracing-logging 2021-04-20 20:16:40 +00:00
Edd Robinson 8f0ed5e51c refactor: satisfy new clippy lints 2021-04-20 17:30:50 +00:00
Marco Neumann c5d99ba9af feat: wire up hard buffer limit
Fixes #1255.
2021-04-20 13:25:31 +02:00
Marco Neumann bada42c0b4 refactor: lower test DB hard limit to 1MB
This makes testing the limit faster and helps with #1255.
2021-04-20 11:43:10 +02:00
Jacob Marble c794340b64 chore: tests check stdout, not stderr 2021-04-19 15:48:30 -07:00
kodiakhq[bot] 15f9c939ef
Merge branch 'main' into refactor-influxdb-client 2021-04-19 17:12:51 +00:00
Carol (Nichols || Goulding) f136931225 fix: Inconsistent ordering lints 2021-04-19 08:48:11 -04:00
Andrew Lamb e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx (#1230)
* refactor: Create Arrow arrays using iterators

* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps

* feat: add support for timestamp array

* fix: update more tests

* fix: remove unecessary code

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
Andrew Lamb f092294da3
fix: Use MAX (window end) for timestamps in read group (#1228)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
Aakash Hemadri 4372da2d58
Merge branch 'main' into refactor-influxdb-client 2021-04-16 10:01:42 +05:30
Aakash Hemadri 966b97903c
fix: ci tests & write module
Add write to api/mod.rs
Rename influx2.rs -> write.rs
Fix end_to_end_cases for DataPoint

Signed-off-by: Aakash Hemadri <aakashhemadri123@gmail.com>
2021-04-16 09:58:23 +05:30
Andrew Lamb 130e059b6b
fix: Flight API should be able to return zero RecordBatches (#1229) 2021-04-15 19:10:51 +00:00
Edd Robinson 9834c845db test: add influxrpc tag_values benches
The initial benchmarks look like this on my i9 MBP:

```
Data in one open chunk and one closed chunk of mutable buffer/tag0/no_pred           1.00     91.0±2.55ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag0/with_pred         1.00     11.5±0.72ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/no_pred           1.00    120.3±5.10ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/with_pred         1.00     11.2±0.22ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/no_pred           1.00    203.2±8.45ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/with_pred         1.00     11.2±0.21ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/no_pred      1.00    100.3±3.73ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/with_pred    1.00     31.2±1.80ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/no_pred      1.00    126.7±2.29ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/with_pred    1.00     33.0±1.70ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/no_pred      1.00    212.0±6.86ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/with_pred    1.00     18.1±0.99ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/no_pred                             1.00     98.7±6.08ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/with_pred                           1.00     11.2±0.37ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/no_pred                             1.00    118.9±3.97ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/with_pred                           1.00     11.7±0.64ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/no_pred                             1.00    202.1±8.49ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/with_pred                           1.00     11.1±0.27ms        ? ?/sec
Data in two read buffer chunks/tag0/no_pred                                          1.00    109.2±5.20ms        ? ?/sec
Data in two read buffer chunks/tag0/with_pred                                        1.00     44.2±1.83ms        ? ?/sec
Data in two read buffer chunks/tag1/no_pred                                          1.00    132.9±3.79ms        ? ?/sec
Data in two read buffer chunks/tag1/with_pred                                        1.00     41.7±2.43ms        ? ?/sec
Data in two read buffer chunks/tag2/no_pred                                          1.00    222.4±7.00ms        ? ?/sec
Data in two read buffer chunks/tag2/with_pred                                        1.00     27.9±0.92ms        ? ?/sec
```
2021-04-14 09:36:39 +00:00
Marko Mikulicic c9cbc74857 feat: Use a DID newtype instead of u32 for dictionary ids
Rationale
---------

We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
Marko Mikulicic e76980928b feat: Implement Update API 2021-04-08 22:25:36 +00:00
Raphael Taylor-Davies ef2dc952e1
feat: write once writer id (#1131)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 17:51:57 +00:00
Raphael Taylor-Davies b0e21e5f9e
feat: automated chunk lifecycle (#1091)
* feat: automated chunk lifecycle

* chore: use >= for lifecycle timestamp comparisons

* chore: review fixes
2021-04-01 11:05:58 +00:00
Andrew Lamb 7154dfd5f6
feat: Add timestamps to ChunkSummary (#1079)
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk

* feat: Add timestamps to ChunkSummary

* feat: Add timestamp conversion logic to protobuf types

* test: Add tests

* fix: Update data_types test

* fix: handle negative nanos during conversion

* fix: clippy

* fix: more clippy

* fix: even more clippy

* fix: even more clippy
2021-03-30 19:03:23 +00:00