5dfd152549
test: Use the now_nanos helper function more in tests
2021-05-12 10:58:08 -04:00
f28c9ae04c
docs: Add unit and semantic information about the process clock
2021-05-12 10:58:08 -04:00
513d4731be
feat: Add a process clock to Db and use it for Sequenced Entries
...
Connects to #1157 .
2021-05-12 10:58:06 -04:00
f98807936d
test: Some tests don't call await, so they don't need to be async
2021-05-12 10:57:05 -04:00
696e4e0cfd
fix: ensure metrics not overwriting
2021-05-11 20:57:31 +01:00
4409d2c8af
feat: instrument catalog locks ( #1464 )
...
* feat: instrument catalog locks (#1355 )
* chore: add metrics test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-11 18:59:11 +00:00
9d0c3a2b1a
refactor: Remove multi-table per chunk code in MUB ( #1471 )
...
* refactor: Remove multi-table per chunk code in MUB
* fix: clippy
* fix: bench build
* fix: merge conflicts
2021-05-11 17:49:07 +00:00
d1da954fe4
feat: don't store encoded strings twice in RLE dictionaries ( #1469 )
2021-05-11 15:22:25 +00:00
3622a92c8b
feat: wire in rb column metrics
2021-05-11 13:00:52 +01:00
795f5bfcb7
refactor: make `StatValues::{min,max}` optional + handle NaNs
...
This will allow us to:
- handle all-NULL columns correctly
- be in-line with Parquet (where min/max are optional)
- handle NaNs at least somewhat sane (they do not "poison" stats
anymore)
2021-05-10 17:12:25 +02:00
f037c1281a
feat: Calculate all system tables "on demand" ( #1452 )
...
* feat: compute system.columns table on demand
* feat: compute system.chunk_columns on demand
* feat: compute system.operations on demand
* fix: fixup schemas
* fix: Log errors
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-10 14:43:55 +00:00
9f5350a6c5
fix: Load only databases for which a config exists
...
Closes #1450
2021-05-10 13:14:22 +02:00
c6b933eb63
chore: merge main to branch
2021-05-07 18:40:17 -04:00
971500681f
refactor: address Andrew's and Carol's comment
2021-05-07 17:33:19 -04:00
ba015ee4df
refactor: clean up and add comments
2021-05-07 09:31:41 -04:00
eae3fec571
feat: wire up regex UDF as predicate filter expr
2021-05-07 13:44:51 +01:00
b5ea71f45f
feat: Expose the storage usage for each column in system.chunk_columns ( #1441 )
...
* feat: Expose the storage usage for each column in system.chunk_columns
* fix: fixup logical conflicts
* refactor: move coalsce logic into the read buffer
* fix: Update system_tables to not use coalese
* fix: Improve comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-07 12:36:49 +00:00
9320f59de0
feat: add shard sink indirection ( #1447 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 11:04:51 +00:00
d7253c72c0
feat: Only calculate system.chunks table "on demand" ( #1446 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-07 10:26:56 +00:00
febc1538ff
chore: Update Rust version ( #1445 )
...
* chore: Update Rust version
* refactor: Make struct constructor field orderings consistent
Sometimes I changed the struct definition, sometimes changed the struct
construction instance, depending on consistency with code around each
(other similar structs, function argument orders, etc)
More info: https://rust-lang.github.io/rust-clippy/master/index.html#inconsistent_struct_constructor
* refactor: Use flatten where appropriate
One instance is a false positive with a clippy bug.
More info:
- https://rust-lang.github.io/rust-clippy/master/index.html#filter_map_identity
- https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten
* refactor: Use Option map instead of match
More info: https://rust-lang.github.io/rust-clippy/master/index.html#manual_map
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 22:07:10 +00:00
55bf848bd2
feat: Now we can query directly from files in object store
2021-05-06 18:02:17 -04:00
7f6b11266d
feat: instrument catalog locks ( #1355 ) ( #1439 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 17:09:52 +00:00
44de42906f
refactor: use Arc<str> instead of Arc<String> ( #1442 )
2021-05-06 17:05:08 +00:00
49c0b8b90c
feat: pull-based metrics ( #1355 ) ( #1414 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:54:30 +00:00
216903a949
refactor: move protobuf conversion logic to generated_types ( #1437 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 15:49:27 +00:00
884baf7329
feat: add column_type and influxdb_column_type, remove row_count from system.columns ( #1415 )
...
* feat: add column_type and influxdb_column_type, remove row_count from system.columns
* fix: update tests
* fix: more test update
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: copy/paste type conversion to avoid cross dependency between data_types and internal_types
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-05-06 12:59:30 +00:00
578dc0db25
feat: Add more logs to shed light on the curious incident with missing metrics in the nighttime
2021-05-06 14:42:48 +02:00
10f89a3e8d
refactor: split entry out into separate crate ( #1428 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 11:36:23 +00:00
a5c92fae8a
chore: merge main to branch
2021-05-05 13:48:42 -04:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
4395ede244
Merge branch 'main' into debug-chunk-metrics
2021-05-05 15:43:32 +00:00
2b0d7cfb91
feat: Add debug to update_chunk_state metrics
2021-05-05 17:37:57 +02:00
fcb37a0b1d
feat: more testing scenarios for quering parquet files
2021-05-05 10:57:02 -04:00
4a64e22e64
refactor: Use trait object and deref instead of cloning Arc in tests
2021-05-05 10:55:12 -04:00
e32fa43a53
docs: Add note about implication of write buffer errors
2021-05-05 10:55:12 -04:00
7d5c988fba
feat: Actually route SequencedEntry to the Write Buffer, if present
...
Connects to #1157 .
Rearrange some code and comments to be consistent with the design. Make
some more places not care whether they're getting an owned or borrowed
SequencedEntry.
2021-05-05 10:55:11 -04:00
54c5f984d5
fix: Use stdlib's path manipulation rather than format
...
The syntax highlighting in my editor broke because of the unmatched
double quote, which got me to look a bit closer at this test. These
tests would have failed on Windows.
2021-05-05 10:55:11 -04:00
231abd221f
refactor: Extract a TestDbBuilder
2021-05-05 10:55:11 -04:00
62dfb47825
refactor: Reorganize test imports
2021-05-05 10:55:11 -04:00
9e61b470e7
feat: change MemoryStream to accept multiple record batches
2021-05-05 13:29:16 +02:00
34754ebcdb
refactor: move MemoryStream to arrow_deps
2021-05-05 13:29:16 +02:00
733d502350
refactor: fix tests
2021-05-04 18:38:42 +01:00
9aa144e0f4
feat: add per-stage current chunk storage
2021-05-04 17:43:53 +01:00
3b7c5ac350
fix(storage rpc): do not send back tags with empty values ( #1403 )
2021-05-04 10:35:24 +00:00
b579ef8646
feat: Add jemalloc stats
2021-05-03 12:10:48 +02:00
3c5595d046
Merge branch 'main' into ntran/unload_chunks
2021-04-30 22:02:38 +00:00
979f5f9347
refactor: write buffer to use sequenced entry and new segment
...
This refactors the write buffer to use the sequenced entry structure and the new segment definition. It removes the old replicated write and write_buffer.fbs.
Finally, it updates the SequencedEntry wrapper type around the Flatbuffer structure to be a trait so that SequencedEntry can be initialized from a borrowed Flatbuffer or an owned Vec<u8>.
How writes go into segments in the buffer and any kind of validation will likely have to be updated based on what kinds of guarantees we want to make in the buffer. However, that should probably come after we've rethought the design a bit around the new layout of chunks in the Parquet persistence.
2021-04-30 17:00:23 -04:00
a967ebfabd
refactor: rename closing to closed ( #1396 )
...
* refactor: rename closing to closed
* refactor: further renames
2021-04-30 20:59:45 +00:00
34a3388a49
feat: unload chunks from read buffer but keep them in object store
2021-04-30 16:12:02 -04:00
c2f7e7efea
feat: warn on dropping from open partition ( #1395 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 18:56:49 +00:00
ebe671e59e
refactor: include default labels
2021-04-30 14:03:09 +01:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
c9b33c6b7d
chore: Merge branch 'main' into ntran/test_query_parquets
2021-04-29 14:22:34 -04:00
f89d56af8a
chore: merge mian to branch
2021-04-29 14:14:04 -04:00
2d643d4407
fix: set nullability correctly for system tables ( #1387 )
2021-04-29 18:09:27 +00:00
e87973babe
refactor: address review comments
2021-04-29 13:15:43 -04:00
ea476c86be
feat: add chunk size histogram
2021-04-29 17:10:57 +01:00
5ef9be4abd
refactor: use RecordBatch::try_from_iter to make system tables ( #1374 )
...
* refactor: use RecordBatch::try_from_iter to make system tables
* fix: clippy
2021-04-29 15:32:35 +00:00
0788892413
feat: add row_count to system.chunks and Chunk managment API ( #1373 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-29 13:53:25 +00:00
3672a20570
feat: Add ingest_entries_bytes_total counter
2021-04-29 14:03:05 +02:00
262bf446ab
feat: change background worker to use backoff instead of poll ( #1339 ) ( #1347 )
...
* feat: change background worker to use backoff instead (#1339 )
* chore: fix comment
* chore: fix tests
* chore: review comments
* chore: review feedback
2021-04-29 11:10:51 +00:00
b1924d10b4
fix: Only show current database's system.operations table ( #1310 )
...
* test: add test for operations.system_tables
* fix: only show operations for current database
* fix: update test
* fix: improve test
* refactor: filter in Schema provider rather than in job tracker
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 21:06:19 +00:00
402d9c748c
chore: cargo fmt
2021-04-28 16:52:52 -04:00
0ea8861a65
chore: fix merge that causes test failures
2021-04-28 16:38:29 -04:00
2a2760bd18
feat: complete tests where data in both RUB and OS
2021-04-28 16:14:07 -04:00
140d96dbea
feat: tests ffor loading data to object store and make sure twe still query read buffer
2021-04-28 15:59:17 -04:00
9aefcd216f
fix: Validate that ClockValue is never 0
2021-04-28 13:54:55 -04:00
dcc732b154
Merge branch 'main' into cn/server-id-fix
2021-04-28 17:19:29 +00:00
2f4d7189ff
fix: Validate ServerId when creating structs from flatbuffers
...
When we get the flatbuffers, we won't have the server ID in addition to
the flatbuffers-- it's in the flatbuffers. But we want to validate the
`ServerId` once when the `SequencedEntry` is created so that its
`server_id` method can assume it has a valid `ServerId`.
2021-04-28 13:06:12 -04:00
c4259d5336
refactor: update benchmarks
2021-04-28 16:11:57 +00:00
a9ef604ef6
perf: avoid using channels for query execution
...
Pre-sized channels get full when the results to send over them are larger than the capacities. This causes significant runtime overhead and slows down query performance.
This commit removes the intermediate channels. The potential downside to this approach is there may be more buffering which could increase memory usage during query and also block a thread for longer periods of time.
2021-04-28 16:11:57 +00:00
d6c0f40630
test: add read_group benchmark
2021-04-28 16:11:57 +00:00
5dfe312867
test: add benchmark for influxrpc read_filter
2021-04-28 16:11:57 +00:00
b6054f9df9
feat: Make background task period configurable
2021-04-28 15:47:31 +02:00
9b57214344
feat: move closing chunks as soon as possible ( #1329 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-28 07:50:15 +00:00
6bdc153361
feat: sort RUB (read buffer) chunks ( #1308 )
...
* feat: sort chunks before upserting to read buffer (#1216 )
* chore: review feedback
* chore: fix merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 16:50:48 +00:00
7ca1da3fcd
feat: pushdown table and partition key predicates to catalog ( #736 ) ( #1327 )
...
* feat: catalog predicate pushdown (#736 )
* chore: fix lints
* chore: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 15:31:47 +00:00
91bccdfca3
ci: pass `--document-private-items` to `cargo doc`
2021-04-27 15:42:07 +02:00
e78a81ee59
Merge branch 'main' into crepererum/ci_doc
2021-04-27 12:03:04 +00:00
c3cb86fc1a
Merge branch 'main' into crepererum/issue1296
2021-04-27 11:58:10 +00:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
20117de078
feat: string dictionary encoding ( #1220 ) ( #1262 )
...
* feat: string dictionary encoding (#1220 )
* chore: review comments
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
c8594ce743
refactor: chunks shall always have data (= 1 table)
...
Chunks now always have "data" (aka exactly 1 table including
schema/columns). Open chunks can only be created with data. Rollovers do
NOT create open chunks anymore (this is now only done for incoming
data).
Closes #1296 .
2021-04-27 10:28:57 +02:00
35bc09ef7c
refactor: use `FromIter` rather than array builders to create system table arrays ( #1246 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 19:47:11 +00:00
272cdb85ce
fix: Use the ServerId type everywhere, for writing, querying, anything
2021-04-26 18:44:32 +00:00
b8face3335
refactor: Organize use statements
2021-04-26 18:44:32 +00:00
67f5ad841d
refactor: Introduce ServerId and CurrentServerId types
2021-04-26 18:44:32 +00:00
0a835436ac
feat: use bitmasks within MUB ( #1274 ) ( #1289 )
...
* feat: use bitmasks within MUB (#1274 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-26 18:00:16 +00:00
e33af0c084
fix: remove wait_for_complete
2021-04-26 16:59:36 +00:00
fcf5c304a6
fix: fixup merge conflicts
2021-04-26 16:59:36 +00:00
348f38c2c9
feat: add status column
2021-04-26 16:59:36 +00:00
9c74074095
feat: List background operations as system.operations table
2021-04-26 16:59:36 +00:00
faec98eab9
refactor: remove time column from row group
2021-04-26 09:51:06 +00:00
dd2bee3343
fix: don't over count new chunks
2021-04-23 20:25:37 +01:00
552409ea70
feat: add metrics for chunk lifecycle
2021-04-23 17:13:43 +00:00
0e3b7bff49
refactor: plumb metrics registry to catalog
2021-04-23 17:13:43 +00:00
eacfe9af6a
refactor: address PR feedback
2021-04-23 15:58:48 +00:00
d3218802ab
refactor: more API metric coverage
2021-04-23 15:58:48 +00:00
97b2369140
refactor: swap existing metrics for THE NEW WAY
2021-04-23 15:58:48 +00:00
f12f1eeef7
refactor: make chunks per-table
...
This changes the hierarchy from
```
database -> partition -> chunk -> table
```
to
```
database -> partition -> table -> chunk
```
Only the high-level APIs are changed for now. The chunk states (like
MutableBuffer and ReadBuffer) still multiplex tables, although they will
always only get a single table assigned (or no table if no data was
presented yet).
Closes #1256 .
2021-04-23 09:18:55 +02:00
74c25f541d
feat: fast MUB dictionary arrow conversion ( #1273 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-22 20:31:44 +00:00
83d6550316
feat: Implement write_entry_downstream
2021-04-21 20:50:46 +00:00
88ca1a5245
fix: Rename wal.fbs to write_buffer.fbs
2021-04-21 17:43:03 +00:00
80995afb70
fix: Change WAL to Write Buffer in comments and documentation
2021-04-21 17:43:03 +00:00
5f024401a4
refactor: Rename data_types::wal module to data_types::write_buffer
2021-04-21 17:43:03 +00:00
46f9360c58
fix: Rename WAL_DIR to WRITE_BUFFER_DIR and redefine it as wb, not wal
2021-04-21 17:43:03 +00:00
8baeaab8ca
refactor: Rename Db.wal_buffer to write_buffer
2021-04-21 17:43:03 +00:00
4bfcb68536
refactor: Remove a WalError that doesn't appear to be used
2021-04-21 17:43:03 +00:00
0677c544bb
refactor: Rename WalBufferRollover to WriteBufferRollover
2021-04-21 17:43:03 +00:00
ee45db402b
refactor: Rename WalBufferConfig to WriteBufferConfig
2021-04-21 17:43:03 +00:00
1cf3ab3978
feat: enable disabling MUB snapshot caching ( #1203 ) ( #1264 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-21 08:21:31 +00:00
508aa8be8c
refactor: satisfy new clippy lints
2021-04-20 17:30:50 +00:00
c5d99ba9af
feat: wire up hard buffer limit
...
Fixes #1255 .
2021-04-20 13:25:31 +02:00
d05585f736
refactor: log chunk migration to rb at info
2021-04-19 16:08:49 +00:00
82c1d94ce1
refactor: Use Option.map where possible
2021-04-19 08:48:11 -04:00
fd0da7e74a
chore: upgrade arrow and Rust
...
See https://github.com/apache/arrow/pull/10082 for upstream PR.
2021-04-19 14:00:04 +02:00
4c23ca8888
feat: full implementation of parquet's read_filter for review
2021-04-16 16:03:24 -04:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
f092294da3
fix: Use MAX (window end) for timestamps in read group ( #1228 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
878b1b318e
feat: Initial scaffolding for routing layer
...
Part of #916
Adding first-class concept of ShardId in shard config, fixes #1156
NEXT:
- [ ] implement sharder
- [ ] implement `write_entry_downstream`
- [ ] add tests
2021-04-15 09:02:47 +00:00
4001ec73c9
Merge branch 'main' into ntran/query_parquet_2
2021-04-14 20:33:03 +00:00
4e2d59d9a5
feat: saimplement a few more functions as part of supporting query dfrom parquet files
2021-04-14 16:06:47 -04:00
025f760479
refactor: change sync::RwLock to parking_lot
2021-04-14 19:18:03 +00:00
59ca090aef
feat: Use single db-wide executor for running queries ( #1198 )
...
* refactor: plumb executor into all Db instances
* refactor: Route all query executions through worker pool
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 16:46:02 +00:00
5bb34e9a97
refactor: use read_buffer column range for time range
2021-04-14 16:10:24 +00:00
04f594bf94
perf: use column_values in read buffer
2021-04-14 15:13:35 +00:00
f5f768d750
feat: Add a dedicated threadpool for running queries ( #1191 )
...
* feat: use a dedicated tokio threadpool for running queries
* feat: plumb number of executor threads through to command line
thread through command line
* fix: Logical merge conflict
* fix: another logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 10:48:09 +00:00
f430013141
refactor: rename `Db::load_chunk_to_object_store` to `Db::write_chunk_to_object_store
...
For #1083 .
2021-04-14 12:37:43 +02:00
1c6690ae3b
fix: fix constant use in `server::db::tests::read_from_read_buffer`
2021-04-14 12:37:43 +02:00
26ce520cf1
feat: hook RB->OS into lifecycle
...
Fixes #1083 .
2021-04-14 12:37:43 +02:00
7b0b975433
feat: add Db::load_chunk_to_object_store_in_background
...
For #1083 .
2021-04-14 12:37:43 +02:00
84b2ce4571
docs: extend docs of `Db::load_chunk_to_read_buffer_in_background`
2021-04-14 12:37:43 +02:00
9834c845db
test: add influxrpc tag_values benches
...
The initial benchmarks look like this on my i9 MBP:
```
Data in one open chunk and one closed chunk of mutable buffer/tag0/no_pred 1.00 91.0±2.55ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag0/with_pred 1.00 11.5±0.72ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/no_pred 1.00 120.3±5.10ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag1/with_pred 1.00 11.2±0.22ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/no_pred 1.00 203.2±8.45ms ? ?/sec
Data in one open chunk and one closed chunk of mutable buffer/tag2/with_pred 1.00 11.2±0.21ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/no_pred 1.00 100.3±3.73ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag0/with_pred 1.00 31.2±1.80ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/no_pred 1.00 126.7±2.29ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag1/with_pred 1.00 33.0±1.70ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/no_pred 1.00 212.0±6.86ms ? ?/sec
Data in open chunk of mutable buffer, and one chunk of read buffer/tag2/with_pred 1.00 18.1±0.99ms ? ?/sec
Data in single open chunk of mutable buffer/tag0/no_pred 1.00 98.7±6.08ms ? ?/sec
Data in single open chunk of mutable buffer/tag0/with_pred 1.00 11.2±0.37ms ? ?/sec
Data in single open chunk of mutable buffer/tag1/no_pred 1.00 118.9±3.97ms ? ?/sec
Data in single open chunk of mutable buffer/tag1/with_pred 1.00 11.7±0.64ms ? ?/sec
Data in single open chunk of mutable buffer/tag2/no_pred 1.00 202.1±8.49ms ? ?/sec
Data in single open chunk of mutable buffer/tag2/with_pred 1.00 11.1±0.27ms ? ?/sec
Data in two read buffer chunks/tag0/no_pred 1.00 109.2±5.20ms ? ?/sec
Data in two read buffer chunks/tag0/with_pred 1.00 44.2±1.83ms ? ?/sec
Data in two read buffer chunks/tag1/no_pred 1.00 132.9±3.79ms ? ?/sec
Data in two read buffer chunks/tag1/with_pred 1.00 41.7±2.43ms ? ?/sec
Data in two read buffer chunks/tag2/no_pred 1.00 222.4±7.00ms ? ?/sec
Data in two read buffer chunks/tag2/with_pred 1.00 27.9±0.92ms ? ?/sec
```
2021-04-14 09:36:39 +00:00
4db3a4b3b5
test: enable writer to split large batches
2021-04-14 09:36:39 +00:00
3cfbfe8ebf
refactor: expose query_tests to benches crate
2021-04-14 09:36:39 +00:00
8e0ee48018
Merge branch 'main' into ntran/query_local_parquet
2021-04-13 22:38:56 +00:00
c9cbc74857
feat: Use a DID newtype instead of u32 for dictionary ids
...
Rationale
---------
We use `u32` throughout the codebase to reference for interned dictionary strings.
We also use `u32` for other reasons and it would be nice to get some help from the compiler
to avoid mixing them up
2021-04-13 22:38:41 +00:00
05bf28ce85
feat: Add 2 main functions table_schema and table_names for Parquet Chunk ato pay a foundation for querying it
2021-04-13 18:23:55 -04:00
150ed4e1d9
refactor: Remove async from `InfluxRPCPlanner` ( #1200 )
...
* refactor: Remove async from InfluxRPCPlanner
* fix: make it compile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 22:17:19 +00:00
59c35e5a4f
feat: Implement WriteBuffer gRPC
2021-04-13 21:43:54 +00:00
61cd745ab6
refactor: remove mutable buffer predicate logic ( #1186 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 19:09:36 +00:00
4a6d6bd7ad
feat: initial work for querying data from parquet file in object store
2021-04-13 13:57:46 -04:00
bd13c09bad
refactor: make sharder optional when generating entry
2021-04-13 12:52:14 +00:00
4d22982b5f
fix: fix broken tests from column refactoring
2021-04-13 12:52:14 +00:00
7e28f8ef66
feat: Implement Entry writing to Db
...
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.
I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
494480bc57
fix: check filename to have deterministic output
2021-04-12 15:37:08 -04:00
7f77a01e61
chore: merged main to branch and resolved conflicts
2021-04-12 12:09:04 -04:00
e86a02efa7
refactor: Address comments and add more tests
2021-04-12 11:46:31 -04:00
453aeaf1a0
feat: Add tests for writing RB chunks to Object Store
2021-04-09 17:39:23 -04:00
078c0f3fda
refactor: lift chunk and table summaries out of DBChunk ( #1162 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 12:00:47 +00:00
e76980928b
feat: Implement Update API
2021-04-08 22:25:36 +00:00
738134004e
feat: implement memory tracking on read buffer
2021-04-08 18:20:37 +00:00
bd3816a6c9
chore: merge main to branch
2021-04-08 10:59:33 -04:00
d429cf9aeb
refactor: tighten up Read Buffer API
2021-04-08 10:24:19 +00:00
dae9f12593
refactor: remove deprecated API
2021-04-08 10:24:19 +00:00
232d6ec439
refactor: appease clippy gods
2021-04-08 10:19:11 +00:00
bd21cf5d24
refactor: remove read buffer db from db
2021-04-08 10:19:11 +00:00
17853266ce
refactor: add helper method for all names
2021-04-08 10:19:11 +00:00
c32fe3decd
chore: clean up
2021-04-07 21:43:45 -04:00
f501a74aea
refactor: Address review comments
2021-04-07 21:28:03 -04:00
be6e1e48e4
feat: add writer_id and object_store in Db
2021-04-07 18:36:07 -04:00
319e66706e
Merge branch 'main' into cn/no-more-test-results
2021-04-07 16:45:32 +00:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
c2355aca6d
feat: add basic memory tracking ( #1125 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-07 15:38:24 +00:00
3769ad3d21
test: update test
2021-04-07 10:46:08 +00:00
d4c090cb84
refactor: extract OnceNonZeroU32 ( #1134 )
2021-04-06 21:38:04 +00:00
864f9bcd35
fix: avoid spamming the logs with errors that are not errors ( #1129 )
...
* fix: avoid spamming the logs with errors that are not errors
* fix: make it compile
* fix: change to warning, add database name
* fix: include buffer_size and soft_limit as well
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 20:21:56 +00:00
ef2dc952e1
feat: write once writer id ( #1131 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 17:51:57 +00:00
5cd1d6691d
refactor: use DatabaseName in DatabaseRules ( #1127 )
2021-04-06 13:26:30 +00:00
7cc9f06e74
chore: Update arrow / datafusion deps again ( #1126 )
...
* chore: Update arrow dependencies
* test: add test for SHOW COLUMNS
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-06 12:41:41 +00:00
9a2e636d8c
refactor: move tracking utilities into separate crate ( #1124 )
2021-04-06 11:43:11 +00:00
6e01fbc382
feat: ause TableSummary as metadata for parquet chunk's tables and read buffer's read_filter ot get data
2021-04-05 15:37:34 -04:00
59bd66a33a
refactor: Merge remote-tracking branch 'origin' into ntran/write_parquet_3
2021-04-05 09:40:08 -04:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
4bdf8963e6
feat: continue buidling foundation for writing RB chunks to parquet files
2021-04-02 16:06:25 -04:00
5a04c85da5
Merge branch 'main' into ntran/parquet_write_2
2021-04-02 14:43:46 +00:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
eba799e4e7
Merge branch 'main' into ntran/parquet_write_2
2021-04-01 21:54:19 +00:00
839fe514a2
fix: do not make rows for empty partitions or empty tables
2021-04-01 21:22:04 +00:00
18ca5c5292
refactor: Implement `as_str()` directly rather than using a crate
2021-04-01 21:22:04 +00:00
e52c6a8292
test: Add / update query tests
2021-04-01 21:22:04 +00:00
d0e901f503
chore: fix row_estimate to match comment
2021-04-01 21:22:04 +00:00
c0abeba792
feat: system tables
2021-04-01 21:22:04 +00:00
f66c67692c
feat: add 2 more chunk states for writing chunks to object store
2021-04-01 17:04:08 -04:00
49267114d3
chore: merge main into branch and resolve conflicts
2021-04-01 13:22:49 -04:00
1463c6645f
feat: Add ChunkState::ObjectStore and rename ParquetChunk to Chunk
2021-04-01 11:53:03 -04:00
3a150594ab
refactor: push chunk and partition summary into catalog ( #1103 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-01 13:34:21 +00:00
a59a6edbb8
feat: add SHOW TABLES and `select * from information_schema.columns` ( #1102 )
...
* chore: update dependencies
* test: add tests for new information schema
2021-04-01 12:50:37 +00:00
0d599eeaf4
refactor: remove Db::writeable ( #1101 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-01 11:45:40 +00:00
b0e21e5f9e
feat: automated chunk lifecycle ( #1091 )
...
* feat: automated chunk lifecycle
* chore: use >= for lifecycle timestamp comparisons
* chore: review fixes
2021-04-01 11:05:58 +00:00
8f69a12ec6
refactor: Add rwlock around db rules
2021-04-01 10:43:46 +00:00
de25dc1b53
refactor: remove stuff moved to different crate
2021-03-31 20:02:34 -04:00
19a453a483
feat: finally have some framework with clear todos for writing a chunk into parquet files
2021-03-31 16:21:53 -04:00
b61875e0b2
feat: Add TableSummary calculation to ReadBuffer ( #1092 )
2021-03-31 16:26:37 +00:00
569f90d937
feat: Add ability to get PartitionSummary statistics from a Db ( #1090 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-31 14:18:53 +00:00
13612047c0
feat: chunk sorting ( #1077 )
2021-03-31 12:30:59 +00:00
cd409b471f
feat: continue the implementation
2021-03-30 21:31:51 -04:00
bccac45c75
chore: Merge remote-tracking branch 'origin' into ntran/parquet_write
2021-03-30 18:31:32 -04:00
0bcd52d5c9
feat: Add more changes
2021-03-30 18:31:09 -04:00
7154dfd5f6
feat: Add timestamps to ChunkSummary ( #1079 )
...
* refactor: Move timestamps from mutable_buffer::Chunk to catalog::Chunk
* feat: Add timestamps to ChunkSummary
* feat: Add timestamp conversion logic to protobuf types
* test: Add tests
* fix: Update data_types test
* fix: handle negative nanos during conversion
* fix: clippy
* fix: more clippy
* fix: even more clippy
* fix: even more clippy
2021-03-30 19:03:23 +00:00
d8d5b1568a
feat: automated chunk closing ( #1074 )
...
* feat: automated chunk closing
* chore: remove unused error variant
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 15:31:09 +00:00
1a0e698571
feat: initial of write chunks to parquet files
2021-03-30 09:56:53 -04:00
f0b411cd43
feat: enable information_schema
2021-03-30 09:01:43 -04:00
0b7dea09d5
refactor: Remove `MutableBufferDb` and Partition ( #1068 )
...
* refactor: Move test for missing tags as null to db
* refactor: move last_write_at and create_time to catalog partition
* refactor: remove mutable buffer partition and port tests
* fix: port over partition sorting logic
* fix: remove stale comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 12:51:37 +00:00
18f3124721
refactor: remove DbCatalog newtype wrapper ( #1067 )
...
* refactor: remove DbCatalog newtype wrapper
* chore: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-30 07:52:16 +00:00
6a48001d13
refactor: Manage storage directly in the Catalog ( #1057 )
...
* refactor: Manage mutable buffer chunks directly
* fix: do not use mutable_buffer for listing table names
2021-03-29 17:55:07 +00:00
1a4c793270
feat: lifecycle rules ( #1045 )
2021-03-29 15:32:36 +00:00
eb0122655d
refactor: Remove async from PartitionChunk ( #1062 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-29 13:00:36 +00:00
02ae743e8e
refactor: Remove async from Database ( #1063 )
2021-03-29 12:48:12 +00:00
fb130ea99d
feat: use CatalogProvider and SchemaProvider ( #1058 )
...
* feat: use CatalogProvider and SchemaProvider
* refactor: review comments
2021-03-29 11:08:46 +00:00
0ca9ad7285
refactor: Remove async from `PartitionChunk::table_schema` ( #1060 )
2021-03-27 18:08:12 +00:00
790e5d3348
refactor: Add fine grained (object level) catalog locking ( #1053 )
...
* refactor: inline catalog crate to server
* refactor: Add fine grained (object level) catalog locking
* fix: Move mod definition and use to top of file
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-26 13:48:24 +00:00
1b3fa7c5e8
fix: increase timing tolerance for tracker timing test ( #1051 ) ( #1052 )
...
* fix: increase timing tolerance for tracker timing test (#1051 )
* fix: fmt
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2021-03-25 15:09:09 +00:00
cda7d13110
refactor: unify serialization of database rules ( #973 ) ( #1047 )
...
* refactor: unify serialization of database rules (#973 )
* chore: fix test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-25 14:13:14 +00:00
4cbf7c6ae5
refactor: Make `Db::mutable_buffer_chunks` and `Db::read_buffer_chunks` private ( #1044 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 20:57:15 +00:00
44d67db733
feat: Rework Db to use Catalog for chunk state ( #1043 )
...
* feat: Rework Db to use Catalog for chunk state
* docs: Update server/src/db.rs
* fix: fmt
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 17:57:11 +00:00
360df39d4c
feat: per-Db background tasks ( #1032 )
...
* feat: per-Db background tasks (#1007 )
* chore: review comments
* fix: ensure tracker registry is drained on shutdown
* chore: formatting
* chore: further review fixes
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-24 09:43:52 +00:00
d4eef65f2a
feat: share job registry with Db struct ( #1038 )
2021-03-23 12:34:55 +00:00
1cbfea7096
refactor: don't implement serialize for Db ( #1037 )
2021-03-23 11:36:46 +00:00
9e9387352e
Merge branch 'main' into ntran/unsigned_tests
2021-03-22 16:26:08 -04:00
6a2e5d69f1
chore: e2e test for unsigned types
2021-03-22 15:50:42 -04:00
c47a252c15
feat: Check in generated WAL flatbuffers code
2021-03-22 09:38:59 -04:00
52e415ae1b
fix: Verify flatbuffers when restoring from a file, and not after that
2021-03-22 09:38:59 -04:00
24af745b23
chore: Upgrade to flatbuffers 0.8
2021-03-22 09:38:58 -04:00
6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types ( #1015 )
...
* refactor: Move some types (not yet exposed to clients) into internal_types
* docs: Add README.md explaining the rationale
* refactor: remove some stragglers
* fix: fix benches
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add clippy lints
* fix: fmt
* docs: Apply suggestions from code review
fix typos
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
7e6c6d67b4
feat: graceful shutdown ( #827 ) ( #1018 )
...
* feat: graceful shutdown (#827 )
* chore: additional docs
* chore: further docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 15:36:49 +00:00
dd94a33bc7
feat: retain limited tracker history ( #1005 )
2021-03-17 16:32:34 +00:00
3a53923684
feat: Management API + CLI command to close a chunk and move to read buffer ( #1002 )
...
* feat: Management API + CLI command to close a chunk and move to read buffer
* refactor: Less copy-pasta
* fix: track only once, use `let _` instead of `.ok()`
* docs: Apply suggestions from code review
fix comments ( 🤦♀️ for copy/pasta)
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update server/src/lib.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Use DatabaseName rather than impl Into<String>
* fix: Fixup logical merge conflicts
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-17 15:25:27 +00:00
1fee56274b
fix: prevent observing inconsistent tracker states ( #994 )
...
* fix: prevent observing inconsistent tracker states
* feat: further tracker docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-17 14:28:54 +00:00
4b98d19d70
refactor: remove interior mutability from TrackerRegistry ( #1004 )
...
* refactor: remove interior mutability from TrackerRegistry
* fix: don't hold mutex across await point
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-17 12:58:27 +00:00
2f77090ecf
fix: clippy/fmt
2021-03-16 18:16:48 -04:00
ef9c3f3d8f
feat: Management API + CLI command to list chunks in a partition ( #995 )
...
* feat: Management API + CLI command to list chunks in a partition
For ease of use
* refactor: remove unecessary Result
2021-03-16 20:10:55 +00:00
4781317647
feat: Management API + CLI command to create a new chunk (rollover partition) ( #990 )
...
* feat: Management API + CLI command to create a new chunk (rollover partition)
* fix: Update tests/end_to_end_cases/management_api.rs
fix typo
* fix: logical merge conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 16:41:18 +00:00
65f7a1ac5b
fix: use consistent crate versions ( #989 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 15:42:19 +00:00
7e25c4e896
feat: add fanout task tracking ( #956 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-12 15:01:27 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
0ff527285c
refactor: remove unnecessary async from DatabaseStore trait ( #965 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-11 11:33:53 +00:00
1b40ca7ab5
feat: Implement CRUD for remote addresses
2021-03-10 13:39:59 +00:00
706115178b
chore: remove cruft from config/database rules ( #935 )
...
The replication, query, and subscription concepts here are going to be signficiantly different. Thought it would be best to just remove this cruft for now to avoid confusion.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-09 13:48:59 +00:00
a6965769b4
refactor: Remove impl of `query::PartititionChunk` in mutable_buffer: This PR ( #923 )
2021-03-05 10:51:08 +00:00
b1e5cfedf7
refactor: Move chunk predicate creation from query::Predicate into server crate ( #922 )
...
* refactor: Create `ChunkPredicate` predicate in the Server crate
* refactor: move chunk predicate out of chunk
* fix: remove comment in server/src/db/pred.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 23:23:43 +00:00
3be5c26f92
refactor: Remove impl of `query::Database` in mutable_buffer ( #914 )
2021-03-04 22:02:42 +00:00
8b1f100df3
feat: make read_group and read_window_aggregate work across chunks ( #905 )
...
* feat: make read_group and read_window_aggregate work across chunks
* refactor: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Update query/src/frontend/influxrpc.rs
Improve logic and use strings directly
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 17:06:31 +00:00
c23328d57d
refactor: Use `vec![]` instead of inlined strings, for legibility, in tests ( #911 )
...
* refactor: Use vec![] instead of inlined strings, for legibility
* refactor: use standardized names
2021-03-03 15:16:37 +00:00
94bd200e60
refactor: Add Predicate::is_empty() and EMPTY_PREDICATE to avoid unecessary construction ( #891 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-01 21:03:05 +00:00
7d8d00781c
feat: Make read_filter work for mutable buffer and read buffer ( #882 )
...
* feat: port read_filter to InfluxRPCPlanner
* fix: remove commented out vestigal test
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: Update arrow_deps/src/util.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-01 16:50:29 +00:00
18de3bdcab
chore: merge main into branch
...
Merge branch 'main' into ntran/optimize_column_selection
2021-02-26 15:29:43 -05:00
f37e5846aa
feat: fmt auto fix
2021-02-26 14:56:10 -05:00
eb81975151
feat: Optimize Column Selection
2021-02-26 14:28:46 -05:00
12deacd8a0
refactor: move SeriesSetPlans into its own module ( #878 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 23:12:39 +00:00
8fb7651719
feat: Port tag_values to the InfluxRPCPlanner ( #859 )
...
* feat: Port tag_values to the InfluxRPCPlanner
* refactor: merge imports
* refactor: rename column_names to tag_column_names for clarity
* fix: Update query/src/frontend/influxrpc.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: use ensure!
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: less silly whitespace
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: code review comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-24 23:11:22 +00:00
ed7859e182
refactor: rename tag_column_names --> tag_keys in InfluxRPCPlanner ( #860 )
2021-02-23 17:04:53 +00:00
0aee605544
Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction
2021-02-22 14:13:34 -05:00
6e6cc616a0
refactor: Switch to parking_lot::Mutex
2021-02-22 13:51:31 -05:00
c6becbd544
fix: Use structured log output for partition dropping
2021-02-22 12:46:48 -05:00
45f4855d2e
fix: Sort data in tests to avoid spurious failures
...
The implementation of this function is ultimately the keys from a
HashMap, and that order isn't guaranteed. Not sure if the function needs
a stable ordering, so I didn't want to add the performance hit to the
function. Adding it to the test ensures it keeps passing (I saw it fail
a few times locally).
2021-02-22 12:34:08 -05:00
eec13912dd
fix: Update test that calls a fn that's no longe async
2021-02-22 11:06:54 -05:00
cb335810e2
refactor: use new column_name API
2021-02-22 15:32:55 +00:00
4caa1d732f
refactor: make Selection Copy
2021-02-22 15:32:55 +00:00
d0707725cf
Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction
2021-02-22 10:21:59 -05:00
aa09cd3579
fix: remove workaround query code ( #842 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-22 13:29:27 +00:00
fc0c79a84a
Merge branch 'main' into er/feat/read_buffer/string_fields
2021-02-19 23:10:07 +00:00
92eb8b9e85
refactor: make certain Database method sync
...
A couple of methods don't seem to have any await points in their
implementations, so it feels like they could just be `sync`.
2021-02-19 17:14:17 +00:00
536c1724bd
feat: Allow to put streams of unknown length to objectstore
...
Addresses the API aspect of #818
Adds a utility module that helps computing the length of a stream while buffering it
for later replay (in-memory or spilling it in a temporary file).
2021-02-18 16:49:18 +00:00
58bb79eef2
test: enable scenario
2021-02-17 21:17:56 +00:00
9b91e0624c
feat: implement field_columns plan ( #819 )
...
* feat: implement field_columns plan
* fix: fix doc tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-17 20:43:24 +00:00
b705baa223
refactor: apply clippy clone lint
2021-02-16 13:34:56 +00:00
94a93e56ff
feat: implement `tag_keys` in gRPC planner and across mutable buffer ( #795 )
...
* feat: move tag_column_names into rpc planner
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: compile error
* refactor: remove PassThrough error type
* fix: Avoid extra layers of errors in mutable buffer chunk
* fix: use HashMap::get rather than values() and find
* fix: push filtering down to chunk in gRPC planner
* fix: fixup trait bounds to be non-silly
* fix: remove incorrect comment
* fix: remove cruft
* fix: clippy + fmt
* fix: correct comment
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-15 16:47:52 +00:00
dc465e5d02
feat: Add function to check db size and drop partitions
...
Adds functionality to the server Db to check the mutable buffer size and drop partitions based on the database rules.
2021-02-13 17:19:40 -06:00
b8f85967dd
feat: Enable/Disable logging in tests via RUST_LOG environment variable ( #793 )
...
* feat: Enable/Disable logging in tests via RUST_LOG environment variable
* docs: Add section to contributing
* docs: tweak readme
* fix: Use same logging system in tests as in influxdb_ioxd
2021-02-12 13:43:12 +00:00
a03598dfe2
feat: Implement Cross Chunk Schema / RecordBatch merging at query time ( #783 )
...
* feat: feat: Implement Cross Chunk Schema / RecordBatch merging at query time
* docs: update comments about NullArray::new_with-type
* docs: Update comments based on code review
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 18:26:38 +00:00
7debe94ee6
feat: add background task tracking ( #655 )
2021-02-11 10:30:19 +00:00
4c43c311f0
feat: Add database rules configuration for mutable buffer ( #779 )
...
* feat: Add database rules configuration for mutable buffer
* refactor: change all database rules usage to use mutable_buffer_config rather than store_locally
2021-02-10 18:36:29 -05:00
46dfd58e6e
test: add a SQL test case for selection (of columns) ( #778 )
...
* test: add a SQL test case for selection (of columns)
* test: test with selection + pred
2021-02-10 12:09:19 -05:00
a316b16960
feat: Change table_names to return either Some(set) or None, rather than a plan (try 2) ( #776 )
...
* feat: Change table_names to return either Some(set) or None, rather than a plan
* docs: improve comments
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: merge conflict
* fix: don't clone a string unless needed
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-09 12:20:59 -05:00
143488fae9
feat: add WAL metadata endpoint ( #724 )
2021-02-08 16:21:34 +00:00
8399c56587
feat: remove RwLock on entire ReadBuffer ( #761 )
2021-02-05 16:58:17 -05:00
74b8e8ec7d
refactor: Break Database query tests into their own files + add basic SQL tests ( #760 )
...
* refactor: break query tests into separate modules
* test: Add basic sql tests
2021-02-05 16:04:41 -05:00
47bc28460e
refactor: rename partition, table, and column in parition_meta for clarity ( #757 )
...
* refactor: rename partition, table, and column in parition_meta for clarity
2021-02-05 08:00:22 -05:00
3ec483b769
refactor: Reduce async in mutable buffer, use std::sync ( #749 )
...
* refactor: Reduce async in mutable buffer, use std::sync
* fix: logical confict with new code
2021-02-05 06:47:40 -05:00
fbf776c6b3
chore: Clean up Cargo.tomls ( #754 )
...
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
202a4187a1
feat: Implement list databases API
...
Ref #722
2021-02-04 16:50:40 +01:00
d66eae1a44
feat: Implement TableProvider for Trait for `Db` ( #730 )
...
* feat: Implement TableProvider for Db
Gets us selection pushdown in plans, sets us up for predicate pushdown
Includes: SendableRecordBatchStreams for mutable buffer and read buffer results
fixup snapshots
* docs: comments
2021-02-03 14:18:47 -05:00
8975b1cbf5
fix: into_iter in the assert_table_eq macro
...
This makes it so the macro can take `&[String]` or `&[&str]`.
2021-02-03 09:56:56 -05:00
5c8b351f57
fix: Address clippy suggestions
2021-02-01 14:56:49 -05:00
ff6955a433
refactor: Extract a trait for ObjectStoreApi with associated path
...
This is the promised cleanup. This structure gets rid of a lot of
intermediate structures and encodes through associated types how the
object stores and path types are related.
The enums are still necessary to avoid having generics leak all over
the place, but the object store variants and path variants should always
match because they'll always come from the object store trait
implementations that use the associated types.
2021-02-01 14:56:47 -05:00
7d3b4db234
fix: InMemory doesn't need pagination
2021-02-01 14:35:47 -05:00
fdbe602e57
refactor: Always get a path to build from the object store
2021-02-01 14:30:21 -05:00
5e74ae2ee7
refactor: Segments should know how to persist themselves to object storage
2021-02-01 14:30:20 -05:00
288861e646
feat: implement table_schema in partition chunk, mutable buffer, read buffer ( #705 )
...
fix: sort output schema by name
fix: Update data_types/src/schema.rs
Co-authored-by: Edd Robinson <me@edd.io>
refactor: Update read_buffer/src/lib.rs
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2021-02-01 13:54:58 -05:00
f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) ( #707 )
...
* chore: Update arrow + tokio deps
* chore: Use bleeding edge azure
* chore: Update aws + other deps
* fix: fmt
* fix: Switch to in-house version of routerify
* fix: Upgrade to hyper 0.14
The hyper::error module is now private; hyper::Error is the public
re-export
* fix: Upgrade cloud storage to get tokio upgrade
* fix: Upgrade open_telemetry
* fix: Do not call `panic::set_hook` during another panic
Doing so leads to a double panic which aborts the process.
* fix: new h2 error who dis
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
2282a68e65
refactor: Move selection to the data_types crate and remove redundant implemenation ( #704 )
2021-01-29 13:35:07 -05:00
efb1e0f8ae
feat: Add selection interface to mutable buffer and query interface ( #700 )
...
* feat: Add selection interface to mutable buffer and query interface
* docs: Update mutable_buffer/src/table.rs
* refactor: rename for consistency
* refactor: use map and filter_map rather than fold
2021-01-27 14:31:10 -05:00
df1367bcd1
feat: gRPC measurement_names support for read buffer ( #697 )
...
* test: Add tests for reading data from read buffer in different scenarios
* feat: gRPC measurement_names support for read buffer
2021-01-26 13:15:56 -05:00
504ca67532
test: revamp rpc query testing so it works in multiple chunk scenarios ( #696 )
...
* test: revamp testing so it works in multiple scenarios, fix bug found by same
* fix: Update docs in server/src/db.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: use tsp rather than different functions
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-01-25 16:34:19 -05:00
f7fdadc3b3
Merge branch 'main' into pd-load-database-configs
2021-01-25 09:50:16 -05:00
c3b0371c84
feat: Initial RPC Query Frontend ( #692 )
...
* feat: Initial RPC Query Frontend
* docs: s/immutable buffer/mutable buffer
* docs: Correct type in docstring
2021-01-25 08:33:39 -05:00
b3454daf70
chore: pr fixup
2021-01-22 16:06:10 -05:00
45d25fcbcd
feat: add loading of database rules on startup
...
This adds functionality to the server to load database rules on startup. Follow on work will update the rules to store additional data (the catalog) and ensure that updates to the catalog can occur as outlined in #651 . This work also updated the configuration to not require a database directory so the server can run entirely in memory. I needed this to get the end-to-end test passing since the file object store API doesn't yet have the functionality needed. I've logged #688 to track adding that in.
2021-01-22 16:06:10 -05:00
Carol (Nichols || Goulding)
Edd Robinson
Raphael Taylor-Davies
Andrew Lamb
Marco Neumann
Marko Mikulicic
Nga Tran
Carol (Nichols || Goulding)
kodiakhq[bot]
Paul Dix
Jake Goulding
Andrew Lamb
Jacob Marble
Raphael Taylor-Davies
Paul Dix
NGA TRAN