Commit Graph

340 Commits (44e134792b8b5394ffdc2b95a6cb5484911d8fae)

Author SHA1 Message Date
kodiakhq[bot] ce568b4b2f
Merge branch 'main' into crepererum/issue1253 2021-05-03 11:11:42 +00:00
Marko Mikulicic b579ef8646
feat: Add jemalloc stats 2021-05-03 12:10:48 +02:00
Marco Neumann 136c35cb88 feat: implement transaction handling for catalog
Closes #1253.
2021-05-03 10:04:35 +02:00
Marko Mikulicic 234c899631
feat: Add gauge metric 2021-04-30 14:10:42 +02:00
Edd Robinson 13fbf2e68d refactor: plumb registry to gRPC server 2021-04-29 14:00:05 +01:00
Andrew Lamb 74d35ce9a4
chore: update deps (#1365) 2021-04-29 10:52:43 +00:00
Andrew Lamb a64e622f6e
feat: Add interactive SQL repl (#1332)
* feat: Add interactive SQL repl

* fix: try and fix test

* fix: remove test for prompt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 20:41:02 +00:00
Raphael Taylor-Davies 7ca1da3fcd
feat: pushdown table and partition key predicates to catalog (#736) (#1327)
* feat: catalog predicate pushdown (#736)

* chore: fix lints

* chore: review comments

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 15:31:47 +00:00
Raphael Taylor-Davies 0a835436ac
feat: use bitmasks within MUB (#1274) (#1289)
* feat: use bitmasks within MUB (#1274)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
Raphael Taylor-Davies e393d16e36
chore: update arrow versions (#1300) 2021-04-26 10:11:25 +00:00
Edd Robinson 87de656d23 feat: metrics test helpers 2021-04-23 15:58:48 +00:00
Edd Robinson 97b2369140 refactor: swap existing metrics for THE NEW WAY 2021-04-23 15:58:48 +00:00
Edd Robinson ea909f45ad feat: RED metrics 2021-04-23 15:58:48 +00:00
Raphael Taylor-Davies 74c25f541d
feat: fast MUB dictionary arrow conversion (#1273)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
Raphael Taylor-Davies fe4fa29930
feat: MUB benchmarks (#635) (#1271) 2021-04-22 11:15:32 +00:00
Marko Mikulicic 83d6550316 feat: Implement write_entry_downstream 2021-04-21 20:50:46 +00:00
Carol (Nichols || Goulding) f0b93c5c8c refactor: Rename the wal crate to write_buffer 2021-04-21 17:43:03 +00:00
Raphael Taylor-Davies fdb22c9d9b
chore: update tonic to 0.4.2 (#1268) 2021-04-21 11:54:03 +00:00
kodiakhq[bot] dc6637b448
Merge branch 'main' into jgm-tracing-logging 2021-04-20 20:16:40 +00:00
Edd Robinson 895827aa98 chore: disable SIMD in Arrow 2021-04-20 17:30:50 +00:00
Jacob Marble 5539887cd6 chore: put env_logger back 2021-04-19 15:48:30 -07:00
Jacob Marble 87396edc56 chore: incremental dependency update
This fixes a metrics CI error, and introduces two new CI errors.
2021-04-19 15:48:30 -07:00
Jacob Marble df9dd15ab8 feat(tracing): improve logs and tracing
Logs and traces are emitted via one pipeline. For now, it is not
possible to emit both, but it should be possible in a few weeks, as
tokio/tracing/tracing-subscriber is going through some refactoring recently.

All affected flags are well-documented, and I have tested all but the
OTLP output flags.

chore: clippy happy

chore: revert instrumentation changes

feat: add log format logfmt, log destinations stderr, stdout

chore: clippy happy
2021-04-19 15:48:30 -07:00
Marco Neumann fd0da7e74a chore: upgrade arrow and Rust
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
Edd Robinson 4b706141de refactor: log new row groups added to RB 2021-04-19 10:25:57 +00:00
Nga Tran 4c23ca8888 feat: full implementation of parquet's read_filter for review 2021-04-16 16:03:24 -04:00
Marko Mikulicic 878b1b318e feat: Initial scaffolding for routing layer
Part of #916

Adding first-class concept of ShardId in shard config, fixes #1156

NEXT:

- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
Edd Robinson a3fc5e2474 refactor: change sync::RwLock to parking_lot 2021-04-14 19:18:03 +00:00
Andrew Lamb f5f768d750
feat: Add a dedicated threadpool for running queries (#1191)
* feat: use a dedicated tokio threadpool for running queries

* feat: plumb number of executor threads through to command line

thread through command line

* fix: Logical merge conflict

* fix: another logical conflict

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 10:48:09 +00:00
Edd Robinson 9834c845db test: add influxrpc tag_values benches
The initial benchmarks look like this on my i9 MBP:

```
Data in one open chunk and one closed chunk of mutable buffer/tag0/no_pred           1.00     91.0±2.55ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag0/with_pred         1.00     11.5±0.72ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/no_pred           1.00    120.3±5.10ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/with_pred         1.00     11.2±0.22ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/no_pred           1.00    203.2±8.45ms        ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/with_pred         1.00     11.2±0.21ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/no_pred      1.00    100.3±3.73ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/with_pred    1.00     31.2±1.80ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/no_pred      1.00    126.7±2.29ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/with_pred    1.00     33.0±1.70ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/no_pred      1.00    212.0±6.86ms        ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/with_pred    1.00     18.1±0.99ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/no_pred                             1.00     98.7±6.08ms        ? ?/sec
Data in single open chunk of mutable buffer/tag0/with_pred                           1.00     11.2±0.37ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/no_pred                             1.00    118.9±3.97ms        ? ?/sec
Data in single open chunk of mutable buffer/tag1/with_pred                           1.00     11.7±0.64ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/no_pred                             1.00    202.1±8.49ms        ? ?/sec
Data in single open chunk of mutable buffer/tag2/with_pred                           1.00     11.1±0.27ms        ? ?/sec
Data in two read buffer chunks/tag0/no_pred                                          1.00    109.2±5.20ms        ? ?/sec
Data in two read buffer chunks/tag0/with_pred                                        1.00     44.2±1.83ms        ? ?/sec
Data in two read buffer chunks/tag1/no_pred                                          1.00    132.9±3.79ms        ? ?/sec
Data in two read buffer chunks/tag1/with_pred                                        1.00     41.7±2.43ms        ? ?/sec
Data in two read buffer chunks/tag2/no_pred                                          1.00    222.4±7.00ms        ? ?/sec
Data in two read buffer chunks/tag2/with_pred                                        1.00     27.9±0.92ms        ? ?/sec
```
2021-04-14 09:36:39 +00:00
kodiakhq[bot] 8e0ee48018
Merge branch 'main' into ntran/query_local_parquet 2021-04-13 22:38:56 +00:00
Andrew Lamb 518df742df
chore: update arrow deps (#1195) 2021-04-13 18:05:03 +00:00
Nga Tran 4a6d6bd7ad feat: initial work for querying data from parquet file in object store 2021-04-13 13:57:46 -04:00
Raphael Taylor-Davies 55a77914b1
feat: basic snapshot caching (#1184) 2021-04-13 17:10:28 +00:00
Nga Tran 7f77a01e61 chore: merged main to branch and resolved conflicts 2021-04-12 12:09:04 -04:00
Nga Tran 453aeaf1a0 feat: Add tests for writing RB chunks to Object Store 2021-04-09 17:39:23 -04:00
Jake Goulding 16cc37e7f3
chore: update cloud-storage to 0.9 (#1165)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 14:37:51 +00:00
Marko Mikulicic e76980928b feat: Implement Update API 2021-04-08 22:25:36 +00:00
Carol (Nichols || Goulding) 1552c0113a Merge remote-tracking branch 'origin/main' into feature-query 2021-04-08 14:40:12 -04:00
Edd Robinson a34de76c49 refactor: wire read buffer tracker in 2021-04-08 18:20:37 +00:00
Marko Mikulicic df5349406a feat: Switch to jemalloc 2021-04-08 14:19:35 +00:00
Andrew Lamb a1ee443f2a
chore: update arrow deps (#1148) 2021-04-07 22:08:58 +00:00
Carol (Nichols || Goulding) ebb6bbd13c test: start of integration tests of influxdb2 client against influxdb 2.0 OSS 2021-04-07 14:09:39 -04:00
Raphael Taylor-Davies c2355aca6d
feat: add basic memory tracking (#1125)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-07 15:38:24 +00:00
Andrew Lamb 7cc9f06e74
chore: Update arrow / datafusion deps again (#1126)
* chore: Update arrow dependencies

* test: add test for SHOW COLUMNS

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 12:41:41 +00:00
Raphael Taylor-Davies 9a2e636d8c
refactor: move tracking utilities into separate crate (#1124) 2021-04-06 11:43:11 +00:00
Nga Tran 6e01fbc382 feat: ause TableSummary as metadata for parquet chunk's tables and read buffer's read_filter ot get data 2021-04-05 15:37:34 -04:00
Nga Tran 59bd66a33a refactor: Merge remote-tracking branch 'origin' into ntran/write_parquet_3 2021-04-05 09:40:08 -04:00
Jacob Marble 80d55d0829 chore: rename tracing_deps to observability_deps
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
Nga Tran 4bdf8963e6 feat: continue buidling foundation for writing RB chunks to parquet files 2021-04-02 16:06:25 -04:00