e05ca7f98b
fix: Change a method name that says null to not say null
...
The comment and implementation seem to indicate this is creating
non-null data.
2021-07-12 09:59:12 -04:00
4406d8a219
test: Always initialize a TableSummary on TestChunk
2021-07-12 09:59:12 -04:00
22d4040c81
test: Always initialize a Schema for TestChunk
2021-07-12 09:59:12 -04:00
92cb5986f1
test: Initialize a schema on TestChunk to always exist
2021-07-12 09:59:12 -04:00
78f1c4fc80
test: Chunks can only have one table; no need to specify repeatedly
...
This lets us make the name required and always present on TestChunks,
and make the ID optional.
2021-07-12 09:59:12 -04:00
15aac65c2c
fix: Arrange use statements so rustfmt can manage their order
2021-07-12 09:59:11 -04:00
7b7a60993d
feat: consider time as a special key
2021-07-09 18:54:22 -04:00
8f4463664c
feat: add super_key function
2021-07-09 15:37:04 -04:00
bc958e2ff0
refactor: use Arcs to pass schemas around
2021-07-09 09:45:12 +02:00
09e611deb7
refactor: lift query schema generation up to caller
...
Do no longer scan chunks during query planning to determine the schema
(except for the lifetime jobs where we have a good reason to do so).
Instead pass the schema down to from whoever is triggering the query.
For real SQL queries, we then just use the the table-wide schemas
introduced in #1913 .
Apart from avoiding schema merges we now also don't crash any longer
when no chunks are left in the table (aka columns are present but all
rows are gone).
Fixes #1768 .
Fixes #1884 .
2021-07-09 09:24:21 +02:00
c8126784a8
Merge branch 'main' into ntran/avoid_sort_in_scan
2021-07-08 20:22:18 +00:00
680394b50b
refactor: run fmt
2021-07-08 16:21:42 -04:00
c5733ab4a7
refactor: remove redudant code
2021-07-08 16:11:42 -04:00
6738cb272f
refactor: remove duplicate test
2021-07-08 15:59:25 -04:00
da6249a4df
fix: address reviewers' comments and also fixe a bug they discovered
2021-07-08 15:54:54 -04:00
33bc85ad18
feat: Infrastructure for persistence ( #1925 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 11:14:38 +00:00
7602bde850
chore: Update datafusion deps ( #1799 )
...
* chore: Update datafusion deps + rework code
* refactor: remove workaround as it has been contributed upstream
* fix: Update query/src/exec/split.rs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-08 10:58:32 +00:00
d3c4f8c249
fix: store sort key correctly inthe schema. Update tests to reflect it
2021-07-07 15:55:23 -04:00
e6d995cbd8
chore: Update to Rust 1.53.0 ( #1922 )
...
* chore: Update to Rust 1.53.0
* fix: Update to latest clippy standards
* fix: bad refactor
* fix: Update escaping
* test: update test output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-07 18:02:03 +00:00
76789e5902
feat: store sotkey into the chunk schema of RUB
2021-07-06 17:00:35 -04:00
b6185982f7
refactor: make `ProviderBuilder` a build-time-checked builder
...
It's safer and also avoids cloning / copying state around.
2021-07-06 18:20:05 +02:00
4172d7946c
refactor: make `SchemaMerger` self-consuming
...
The error handling in `merge` was incomplete, aka it could leave the
merger in a half-modified state in case of an error. That's generally a
bad idea and can lead to ugly bugs. Also the "builder" pattern that is
used here usually consumes itself (and provides a clone impl), so it is
easier to reason about modifications. So this commit just changes it to
self-consuming builder.
A nice side effect of the new pattern is also that it is build-time
checked and does not contain a runtime assert any longer.
2021-07-06 18:20:05 +02:00
56c8c8d428
feat: Use separate executor for queries and compactions/moves ( #1870 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:47:50 +00:00
0779b0d9bd
feat: add gRPC listener for new write protocol ( #1842 )
...
* feat: add gRPC listener for new write protocol
* chore: clippy happy
* chore: lint
* chore: cargo fmt --all
* chore: cargo clippy
* chore: protobuf-lint
* chore: more formatting
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 16:15:12 +00:00
e03a1a1def
Merge branch 'main' into ntran/dedup_less_concat
2021-07-01 15:59:22 +00:00
d0afc7a176
refactor: clean up and add a missing else case
2021-07-01 11:00:30 -04:00
5cf623201d
fix: deduplicate the last batch before sending it downstream
2021-07-01 10:45:23 -04:00
7235c7b965
refactor: Remove vestigial execution counters ( #1865 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-01 14:08:06 +00:00
ba919726b6
test: unit tests
2021-06-30 15:01:31 -04:00
2a06b93b00
chore: Merge branch 'main' into ntran/dedup_less_concat
2021-06-30 11:37:15 -04:00
1dbdabd66e
fix: 2 values are also considered to be the same if at least one of them is invalid
2021-06-30 10:52:21 -04:00
62d3305923
feat: optimize the dictionaries in the output of deduplicate node ( #1827 ) ( #1832 )
...
* feat: optimize dedup dictionaries (#1827 )
* fix: handle sliced null bitmasks
* chore: review feedback
2021-06-30 09:30:16 +00:00
e6a4e0d709
refactor: make the code clearer for schema even though they are the same
2021-06-29 17:46:30 -04:00
a249b90952
refactor: refactor and add temp info for debugging
2021-06-29 16:35:50 -04:00
4611e5d584
chore: merge main to branch
2021-06-29 15:39:23 -04:00
388e7b7650
fix: reset last_batch
2021-06-29 15:15:09 -04:00
8f309eb569
feat: improve deduplicate to avoid as many concat_batches as possible
2021-06-29 14:41:54 -04:00
12ae9b012a
refactor: clarify intent of
2021-06-28 17:39:48 +01:00
2e5f10f6b1
feat: Sort the output of split_plans as well ( #1800 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-25 13:02:30 +00:00
4e7cf39b23
chore: Reduce debug logging in query crate ( #1802 )
2021-06-24 21:01:11 +00:00
79446d45be
feat: Implement split_plans ( #1794 )
...
* feat: implement split plan / planner
* fix: Apply suggestions from code review
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* fix: resolve merge conflicts
* fix: add values to panic
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
2021-06-24 18:38:00 +00:00
297fc12db8
feat: compact chunks ( #1776 )
...
* feat: compact chunks
* chore: review feedback
* chore: clippy lints
* chore: document sort key algorithm
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-24 16:49:10 +00:00
0a03605bbc
refactor: pull Channel --> Stream adapater into its own module ( #1793 )
...
* refactor: pull Channel --> Stream adapater into its own module
* docs: Update query/src/exec/stream.rs
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
2021-06-24 10:35:45 +00:00
60eb89cad1
feat: Reorg Planner for merge plans ( #1780 )
...
* feat: Reorg Planner
* docs: add example for split
* fix: clippy
* docs: Specify <= rather than < for split
2021-06-23 10:50:44 +00:00
4c5007f961
fix: Select the correct timestamp for min/max selectors ( #1771 )
...
* test: Reproducer showing that the min/max selectors are order dependent
* fix: pick correct timestamp for first/last selectors
* refactor: remove println
* docs: Fixup comments and add to link to arrow-datafusion/issues/600
* fix: Add debug if timestamp is null
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-22 17:53:54 +00:00
763ade390c
refactor: rename deduplicate --> overlap ( #1779 )
2021-06-22 17:07:53 +00:00
5362c7c924
feat: enable query deduplication ( #1762 )
2021-06-21 18:49:04 +00:00
bed6ec8c31
feat: Handle merging chunks that have different schemas ( #1761 )
...
* feat: Handle merging chunks that have different schemas
* test: print out original (non deduplicated) data in tests
2021-06-21 15:52:13 +00:00
6559a9e997
refactor: use Schema to compute InfluxDB primary keys ( #1757 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-18 21:15:31 +00:00
de67bd3efe
refactor: Remove PartitionChunk::table_schema ( #1756 )
...
* refactor: Remove PartitionChunk::table_schema
* docs: update comments
2021-06-18 16:13:16 +00:00
9beeca3e7c
refactor: Unify schema handling in query crate ( #1755 )
...
* refactor: Unify schema handling in query crate
* fix: doclink
2021-06-18 14:10:57 +00:00
1c13d676b4
refactor: Rename query::PartitionChunk --> query::QueryChunk ( #1754 )
2021-06-18 13:24:09 +00:00
c5eea9af6a
feat: Implement DeduplicateExec ( #1733 )
...
* feat: Implement DeduplicateExec
* fix: Doc comments
* fix: fix comment
* fix: Update with arrow ticket references and use datafusion coalsce batches impl
* refactor: rename inner.rs to algo.rs
* docs: Add additional documentation on rationale for last field value
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Update query/src/provider/deduplicate/algo.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: do not use pub(crate)
* docs: fix test comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-06-17 14:17:52 +00:00
b42218a197
chore: Add proper format for SchemaPivotNode ( #1744 )
2021-06-17 11:32:48 +00:00
38d17a3093
chore: remove unused query dependency ( #1731 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-15 22:06:13 +00:00
e2315f0016
refactor: revert reead_filter debugging
2021-06-14 17:54:21 +01:00
6657e6f596
refactor: update query/src/exec/seriesset.rs
2021-06-14 16:09:02 +01:00
58f4073a7d
Merge branch 'main' into er/fix/dictionary_dupe_keys
2021-06-14 15:59:58 +01:00
ec52bca309
fix: ensure values are different
2021-06-14 15:28:35 +01:00
cf6b658ee3
Merge branch 'main' into er/duplicate_keys
2021-06-14 11:10:45 +00:00
0d8d32fd8f
chore: Update deps to get latest arrow ( #1708 )
...
* chore: Update deps to get latest arrow
* fix: Update to rust 1.52
* fix: clippy
2021-06-14 11:08:09 +00:00
1612ebcbdb
refactor: more debug logging
2021-06-14 12:07:51 +01:00
927d6f890f
Merge branch 'main' into er/duplicate_keys
2021-06-14 10:29:46 +01:00
96fb595cc0
refactor: read_filter debugging
2021-06-14 10:22:05 +01:00
11729b9aa7
test: select non-key from 2 chunks with different key/tag sets ( #1703 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-11 18:52:36 +00:00
736cf1ff6f
Merge branch 'main' into ntran/dedupe_final_union
2021-06-11 09:45:54 -04:00
7dd0416960
refactor: address review comments
2021-06-11 09:43:39 -04:00
e34d157f28
fix: comments
2021-06-11 07:30:49 -04:00
ea9edef716
fix: testing option
2021-06-11 07:18:33 -04:00
fb639ee54f
feat: add UnionExec on top of the scan activities
2021-06-11 07:06:08 -04:00
13dd4b23fd
fix: make pruning debug log less confusing ( #1684 )
2021-06-10 18:35:04 +00:00
16b268402e
Merge branch 'main' into ntran/dedup_merge_exec
2021-06-10 17:13:49 +00:00
46d4ab1f2a
refactor: address review comments
2021-06-10 13:13:02 -04:00
7b1106ff64
chore: enforce `clippy::future_not_send` for `query`
2021-06-10 09:48:35 +02:00
4cf05df35b
feat: hook SortPreservingMergeExec into deduplication framework
2021-06-09 23:29:44 -04:00
4478d900ee
refactor: capture test output
2021-06-09 15:09:13 -04:00
8cc99e3420
Merge branch 'ntran/dedup_within_chunk' of https://github.com/influxdata/influxdb_iox into ntran/dedup_within_chunk
2021-06-09 14:40:29 -04:00
b3c94b9d65
refactor: change order of fields to pass circle CI tests
2021-06-09 14:40:10 -04:00
eed73a30c5
Merge branch 'main' into ntran/dedup_within_chunk
2021-06-09 18:19:17 +00:00
c1c58018fc
refactor: address review comments
2021-06-09 14:17:47 -04:00
89fcc457f4
fix: Fix bug in chunk overlap calculation due to nulls ( #1669 )
...
* fix: Fix bug in chunk overlap calculation due to nulls
* docs: add note about algorithmic complexity
* fix: avoid recursion in normal case
2021-06-09 17:46:39 +00:00
07c4277ca7
refactor: schema merge to give more control over field merging ( #1653 )
...
* refactor: schema merge to give more control over field merging
* chore: review feedback
2021-06-09 06:30:45 +00:00
3d50ff7a60
refactor: remove comments
2021-06-08 21:48:57 -04:00
ab7d3384b7
refactor: remove unused comments
2021-06-08 21:43:02 -04:00
3e10351538
test: add tests for the sort plan
2021-06-08 21:40:46 -04:00
cba7f270b4
docs: Improve comments + whitespace ( #1663 )
2021-06-08 21:13:35 +00:00
68e3a2121f
feat: add SortExec
2021-06-08 15:04:31 -04:00
666204d4a8
fix: remove whitespace changes
2021-06-08 14:46:55 -04:00
b23c4e5210
fix: clippy
2021-06-08 14:44:48 -04:00
fd8a87484e
feat: Hook up chunk grouping into provider
2021-06-08 14:42:37 -04:00
edbf1b7d5e
Merge branch 'main' into ntran/dedup_within_chunk
2021-06-08 13:18:40 -04:00
40cb4f741f
feat: initial implementaton
2021-06-08 13:17:36 -04:00
62e8675737
refactor: move primary_key calculaton to TableSummary ( #1659 )
2021-06-08 17:06:37 +00:00
34ba268cf1
feat: Group chunks by potential overlap ( #1654 )
...
* feat: Group chunks by potential overlap
* docs: clarify in what way the calculation is conservative
* fix: Add test for mixed nulls
2021-06-08 16:55:29 +00:00
b88f277477
feat: enable not eq operator
2021-06-08 15:57:07 +01:00
e9834a907c
feat: Prune on boolean column predicates too ( #1629 )
...
* chore: update deps to get latest DataFusion
* fix: enable boolean pruning tests
* fix: update explain plan tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-07 16:51:30 +00:00
ff641e5638
refactor: address Andrew's comments
2021-06-06 22:36:44 -04:00
2f82a9d670
feat: full foundation for deduplicate with todo functions to finish
2021-06-06 22:09:01 -04:00
ff3215e6a9
feat: Implement Chunk Pruning ( #1567 )
2021-06-04 13:05:22 +00:00
c986ce2c19
feat: Add pruning module to query crate ( #1611 )
...
* feat: Add pruning module
* fix: clippy
* fix: Apply suggestions from code review
* fix: remove erronious claims of DF bugs
* fix: update comments with DF bug reference
2021-06-03 11:07:26 +00:00
e7a97f3ac1
test: merge main and add more tests for deduplicate work
2021-06-02 12:00:40 -04:00
60ad929721
refactor: add macro tto compare output of explains
2021-06-01 16:39:14 -04:00
aa867601e5
chore: merge main with DF plan display fix
2021-06-01 16:17:41 -04:00
d8fbb7b410
refactor: Remove last vestiges of multi-table chunks from PartitionChunk API ( #1588 )
...
* refactor: Remove last vestiges of multi-table chunks from PartitionChunk API
* fix: remove test that can no longer fail
* fix: update tests + code review comments
* fix: clippy
* fix: clippy
* fix: restore test_measurement_fields_error test
2021-06-01 16:12:33 +00:00
d3711a5591
refactor: Use ParquetExec from DataFusion to read parquet files ( #1580 )
...
* refactor: use ParquetExec to read parquet files
* fix: test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-01 14:44:07 +00:00
162a808a8d
refactor: Remove `table_name` from PartitionChunk API ( #1584 )
...
* refactor: Remove `table_name` from PartitionChunk API
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-31 12:05:09 +00:00
d50c7c8919
chore: remove unused dependency ( #1581 )
2021-05-31 09:58:10 +00:00
62147ff0d4
feat: add more explain tests
2021-05-27 12:19:41 -04:00
5d342d7779
feat: associate tracker with lifecycle action ( #1099 ) ( #1556 )
...
* feat: associate tracker with lifecycle action (#1099 )
* chore: docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-27 10:47:35 +00:00
4fcc04e6c9
chore: enable arrow prettyprint feature ( #1566 )
2021-05-27 10:28:14 +00:00
c2fd85209c
feat: wait for task shutdown on DedicatedExecutor ( #1537 )
2021-05-25 11:33:55 +00:00
14ba25f86d
chore: Update datafusion and use released version of arrow crates ( #1546 )
...
* chore: Update datafusion and use released version of arrow crate
* fix: Update for change in API
2021-05-24 15:37:22 +00:00
0563005aac
chore: remove leftover comments
2021-05-21 17:01:49 -04:00
f113abacb5
feat: more unit & e2e tests plus cleanup and addressing review comments of Andrew and Edd
2021-05-21 16:48:43 -04:00
e44a3a87db
feat: fnow predicate is actuallu pushed down to RUB but there are bugs and not working yet
2021-05-20 16:56:15 -04:00
51de37e752
chore: run fmt
2021-05-19 15:28:44 -04:00
11561111d5
chore: merge main to branch
2021-05-19 15:11:15 -04:00
1f13842550
chore: modify comments
2021-05-19 14:49:48 -04:00
087d61f229
feat: Part 1 of predicate push down - Send predicates to MUB, RUB, and Parquet File. Note that MUB has not handled predicates yet
2021-05-19 13:59:51 -04:00
7e223780f3
feat: Implement Display for query::predicate to improve debug printing of plans ( #1519 )
...
* feat: Implement Display for query::predicate to improve debug printing of plans
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-19 12:38:34 +00:00
0680a5167f
chore: Improve DataFusion plan logging ( #1508 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-18 11:08:06 +00:00
07db4932ee
refactor: rename data_types/src/chunk.rs -> data_types/src/chunk_metadata.rs ( #1500 )
2021-05-15 10:18:01 +00:00
8ccc359cab
refactor: address PR feedback
2021-05-07 13:48:44 +01:00
4c4bd2f164
refactor: update query/src/func/regex.rs
2021-05-07 13:44:51 +01:00
4cc7a99854
refactor: include not match in support check
2021-05-07 13:44:51 +01:00
beee3115f4
feat: expose regex =\~ and to gRPC API
2021-05-07 13:44:51 +01:00
eae3fec571
feat: wire up regex UDF as predicate filter expr
2021-05-07 13:44:51 +01:00
3fc2c9fc04
feat: add DataFusion regex match operator
...
This commit adds a new custom UDF to IOx that provide a regex operator to Datafusion plans.
Effectively it allows predicates to contain regex operators that are applied as filters, only allowing rows that satisfy the regex to be returned.
I did not use the Arrow regex kernel for this work because that does not return a boolean array indicating which rows matched a regex, but instead returns a new string array of results. This doesn't work well with DF's approach to filtering.
2021-05-07 13:44:51 +01:00
febc1538ff
chore: Update Rust version ( #1445 )
...
* chore: Update Rust version
* refactor: Make struct constructor field orderings consistent
Sometimes I changed the struct definition, sometimes changed the struct
construction instance, depending on consistency with code around each
(other similar structs, function argument orders, etc)
More info: https://rust-lang.github.io/rust-clippy/master/index.html#inconsistent_struct_constructor
* refactor: Use flatten where appropriate
One instance is a false positive with a clippy bug.
More info:
- https://rust-lang.github.io/rust-clippy/master/index.html#filter_map_identity
- https://rust-lang.github.io/rust-clippy/master/index.html#manual_flatten
* refactor: Use Option map instead of match
More info: https://rust-lang.github.io/rust-clippy/master/index.html#manual_map
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-05-06 22:07:10 +00:00
44de42906f
refactor: use Arc<str> instead of Arc<String> ( #1442 )
2021-05-06 17:05:08 +00:00
411cf134e9
refactor: explode arrow_deps ( #1425 )
...
* refactor: explode arrow_deps
* chore: workaround doctest bug
2021-05-05 16:59:12 +00:00
2f789485e6
refactor: fix spelling
2021-05-05 11:06:04 +01:00
3b7c5ac350
fix(storage rpc): do not send back tags with empty values ( #1403 )
2021-05-04 10:35:24 +00:00
40b9b09cdc
refactor: rename assert_table_eq to assert_batches_eq ( #1368 )
2021-04-30 10:51:08 +00:00
eb8d91cf1c
refactor: remove additional uses of RecordBatch::try_new ( #1378 )
...
* refactor: remove additional uses of RecordBatch::try_new
* fix: fix accidental change
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-30 10:24:47 +00:00
13fbf2e68d
refactor: plumb registry to gRPC server
2021-04-29 14:00:05 +01:00
4acbdcf1c9
refactor: address PR feedback
2021-04-28 16:11:57 +00:00
a9ef604ef6
perf: avoid using channels for query execution
...
Pre-sized channels get full when the results to send over them are larger than the capacities. This causes significant runtime overhead and slows down query performance.
This commit removes the intermediate channels. The potential downside to this approach is there may be more buffering which could increase memory usage during query and also block a thread for longer periods of time.
2021-04-28 16:11:57 +00:00
7ca1da3fcd
feat: pushdown table and partition key predicates to catalog ( #736 ) ( #1327 )
...
* feat: catalog predicate pushdown (#736 )
* chore: fix lints
* chore: review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 15:31:47 +00:00
91bccdfca3
ci: pass `--document-private-items` to `cargo doc`
2021-04-27 15:42:07 +02:00
eddc9319ff
docs: deny broken intradoc links
2021-04-27 13:22:28 +02:00
20117de078
feat: string dictionary encoding ( #1220 ) ( #1262 )
...
* feat: string dictionary encoding (#1220 )
* chore: review comments
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-27 09:36:58 +00:00
a322d05838
refactor: rust fmt
2021-04-20 17:30:50 +00:00
554b3b4662
refactor: satisfy new clippy lints
2021-04-20 17:30:50 +00:00
51041ba2d9
fix: Prefer implementing From over Into
2021-04-19 08:48:11 -04:00
757933afc4
fix: use Self when possible
2021-04-19 08:48:11 -04:00
f136931225
fix: Inconsistent ordering lints
2021-04-19 08:48:11 -04:00
3e87ce5232
fix: Make this trait and methods more idiomatically named
...
"into" usually takes ownership and does a conversion; "as" takes
references and provides a different view.
2021-04-19 08:45:34 -04:00
529c99c93f
fix: don't clone arrays to make TimestampNanosecondArrays ( #1241 )
...
* fix: avoid clone
* fix: remove another clone
2021-04-16 18:40:22 +00:00
e226b5a820
feat: Use TimestampNanosecondArray for timestamps in IOx ( #1230 )
...
* refactor: Create Arrow arrays using iterators
* feat: use Timestamp64(TimeUnit::Nanosecond) for timestamps
* feat: add support for timestamp array
* fix: update more tests
* fix: remove unecessary code
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 15:55:33 +00:00
f092294da3
fix: Use MAX (window end) for timestamps in read group ( #1228 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-16 10:51:38 +00:00
5aeeccb97c
feat: Run query plans on the database wide executor as well ( #1210 )
...
* feat: route all query planning through executor
* fix: Rename JoinError -> TaskJoinError and make message clearer
* fix: remove dangling comment
* fix: remove confusing comments
2021-04-15 11:57:20 +00:00
59ca090aef
feat: Use single db-wide executor for running queries ( #1198 )
...
* refactor: plumb executor into all Db instances
* refactor: Route all query executions through worker pool
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 16:46:02 +00:00
8f1bf8a960
fix: Remove mutex acquisition in impl `std::fmt::Debug` for DedicatedExecutor ( #1205 )
2021-04-14 12:09:40 +00:00
f5f768d750
feat: Add a dedicated threadpool for running queries ( #1191 )
...
* feat: use a dedicated tokio threadpool for running queries
* feat: plumb number of executor threads through to command line
thread through command line
* fix: Logical merge conflict
* fix: another logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-14 10:48:09 +00:00
150ed4e1d9
refactor: Remove async from `InfluxRPCPlanner` ( #1200 )
...
* refactor: Remove async from InfluxRPCPlanner
* fix: make it compile
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-13 22:17:19 +00:00
7e28f8ef66
feat: Implement Entry writing to Db
...
This removes the old ReplicatedWrite structure and implements the writing of an Entry to the Db. I also call out in `server/lib.rs` and in the `Db` where sharding and replication might happen.
I've also added helpers in various places to write line protocol to chunks, tables, and databases. That enabled removing a good amount of code from the test helpers crate.
2021-04-13 12:52:14 +00:00
1997324344
feat: mutable buffer snapshotting ( #1179 )
...
* feat: mutable buffer snapshotting
* chore: review feedback
2021-04-13 12:14:54 +00:00
078c0f3fda
refactor: lift chunk and table summaries out of DBChunk ( #1162 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-04-09 12:00:47 +00:00
be6e1e48e4
feat: add writer_id and object_store in Db
2021-04-07 18:36:07 -04:00
82588d5c72
fix: Don't return Result from test functions
2021-04-07 12:40:00 -04:00
5cd1d6691d
refactor: use DatabaseName in DatabaseRules ( #1127 )
2021-04-06 13:26:30 +00:00
80d55d0829
chore: rename tracing_deps to observability_deps
...
OpenTelemetry makes this necessary.
2021-04-02 13:14:30 -07:00
0b880d3534
chore: Group all tracing-related crates under one crate for easier upgrade management
2021-04-02 09:54:39 -04:00
569f90d937
feat: Add ability to get PartitionSummary statistics from a Db ( #1090 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-31 14:18:53 +00:00
f0b411cd43
feat: enable information_schema
2021-03-30 09:01:43 -04:00
6a48001d13
refactor: Manage storage directly in the Catalog ( #1057 )
...
* refactor: Manage mutable buffer chunks directly
* fix: do not use mutable_buffer for listing table names
2021-03-29 17:55:07 +00:00
eb0122655d
refactor: Remove async from PartitionChunk ( #1062 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-29 13:00:36 +00:00
02ae743e8e
refactor: Remove async from Database ( #1063 )
2021-03-29 12:48:12 +00:00
fb130ea99d
feat: use CatalogProvider and SchemaProvider ( #1058 )
...
* feat: use CatalogProvider and SchemaProvider
* refactor: review comments
2021-03-29 11:08:46 +00:00
0ca9ad7285
refactor: Remove async from `PartitionChunk::table_schema` ( #1060 )
2021-03-27 18:08:12 +00:00
663d4fb6f7
docs: Use Scan rather than InMemoryScan for clarity ( #1049 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-26 14:22:49 +00:00
895e808754
chore: Upgrade arrow deps ( #1046 )
...
* chore: Upgrade dependencies
* chore: upgrade query for new interfaces
* chore: update read_buffer
2021-03-25 13:35:08 +00:00
6e1795fda0
refactor: Move some types (not yet exposed to clients) into internal_types ( #1015 )
...
* refactor: Move some types (not yet exposed to clients) into internal_types
* docs: Add README.md explaining the rationale
* refactor: remove some stragglers
* fix: fix benches
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add clippy lints
* fix: fmt
* docs: Apply suggestions from code review
fix typos
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-19 16:27:57 +00:00
72eff5eed5
chore: update deps (including arrow)
2021-03-16 18:15:44 -04:00
65f7a1ac5b
fix: use consistent crate versions ( #989 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-15 15:42:19 +00:00
6ac7e2c1a7
feat: Add management API and CLI to list chunks ( #968 )
...
* feat: Add management API and CLI to list chunks
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: add comment to protobuf
* fix: fix comment
* fix: fmt, fixup merge errors
* fix: fascinating type dance with prost generated types
* fix: clippy
* fix: move command to influxdb_iox database chunk list
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-12 13:56:14 +00:00
0ff527285c
refactor: remove unnecessary async from DatabaseStore trait ( #965 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-11 11:33:53 +00:00
746373a687
refactor: Remove mutable_buffer crate dependency on query crate ( #927 )
2021-03-05 11:34:27 +00:00
8b1f100df3
feat: make read_group and read_window_aggregate work across chunks ( #905 )
...
* feat: make read_group and read_window_aggregate work across chunks
* refactor: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Update query/src/frontend/influxrpc.rs
Improve logic and use strings directly
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 17:06:31 +00:00
957e05ef25
chore: use newly added Arrow's Expr::is_not_null function
2021-03-03 11:46:49 -05:00
94bd200e60
refactor: Add Predicate::is_empty() and EMPTY_PREDICATE to avoid unecessary construction ( #891 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-01 21:03:05 +00:00
7d8d00781c
feat: Make read_filter work for mutable buffer and read buffer ( #882 )
...
* feat: port read_filter to InfluxRPCPlanner
* fix: remove commented out vestigal test
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: fmt
* fix: Update arrow_deps/src/util.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-01 16:50:29 +00:00
6ad8e1aa33
feat: use newly implemented tags_iter to get Tag columns
2021-02-26 15:54:20 -05:00
18de3bdcab
chore: merge main into branch
...
Merge branch 'main' into ntran/optimize_column_selection
2021-02-26 15:29:43 -05:00
f37e5846aa
feat: fmt auto fix
2021-02-26 14:56:10 -05:00
eb81975151
feat: Optimize Column Selection
2021-02-26 14:28:46 -05:00
12deacd8a0
refactor: move SeriesSetPlans into its own module ( #878 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 23:12:39 +00:00
8fb7651719
feat: Port tag_values to the InfluxRPCPlanner ( #859 )
...
* feat: Port tag_values to the InfluxRPCPlanner
* refactor: merge imports
* refactor: rename column_names to tag_column_names for clarity
* fix: Update query/src/frontend/influxrpc.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: use ensure!
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: less silly whitespace
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: code review comments
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-24 23:11:22 +00:00
ed7859e182
refactor: rename tag_column_names --> tag_keys in InfluxRPCPlanner ( #860 )
2021-02-23 17:04:53 +00:00
0aee605544
Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction
2021-02-22 14:13:34 -05:00
6e6cc616a0
refactor: Switch to parking_lot::Mutex
2021-02-22 13:51:31 -05:00
4caa1d732f
refactor: make Selection Copy
2021-02-22 15:32:55 +00:00
d0707725cf
Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction
2021-02-22 10:21:59 -05:00
92eb8b9e85
refactor: make certain Database method sync
...
A couple of methods don't seem to have any await points in their
implementations, so it feels like they could just be `sync`.
2021-02-19 17:14:17 +00:00
4718b340f3
Merge branch 'main' into ntran/update_arrow_dependencies
2021-02-18 16:20:49 +00:00
213094f8f7
chore: update Arrow dependencies
2021-02-18 10:02:57 -05:00
9b91e0624c
feat: implement field_columns plan ( #819 )
...
* feat: implement field_columns plan
* fix: fix doc tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-17 20:43:24 +00:00
071b13b939
chore: Update dependencies ( #821 )
...
* chore: Update dependencies
* fix: update udf implementation for DataFusion update
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-16 23:27:36 +00:00
94a93e56ff
feat: implement `tag_keys` in gRPC planner and across mutable buffer ( #795 )
...
* feat: move tag_column_names into rpc planner
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: compile error
* refactor: remove PassThrough error type
* fix: Avoid extra layers of errors in mutable buffer chunk
* fix: use HashMap::get rather than values() and find
* fix: push filtering down to chunk in gRPC planner
* fix: fixup trait bounds to be non-silly
* fix: remove incorrect comment
* fix: remove cruft
* fix: clippy + fmt
* fix: correct comment
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-15 16:47:52 +00:00
8eaf006c27
refactor: add arc clone lint
2021-02-15 12:35:21 +00:00
11c97a6235
fix: remove make_null_columns
2021-02-14 05:51:25 -05:00
dc465e5d02
feat: Add function to check db size and drop partitions
...
Adds functionality to the server Db to check the mutable buffer size and drop partitions based on the database rules.
2021-02-13 17:19:40 -06:00
83bfa6d949
feat: Add created_at, last_write_at tracking to partition and sorting
...
This commit adds created_at and last_write_at instants to partitions in the mutable buffer. It adds a method on the mutable buffer database to get back the partitions in sorted order based on either the created_at or last_write_at instants. Ordering based on the summary stats from a column are still left to do.
Finally, it modifies the helper function to create replicated write to take a Partitioner trait that can generate partition keys based on lines, rather than taking the DatabaseRules struct directly. This makes it easier to write test cases where data is split into multiple partitions in the mutable buffer.
2021-02-13 17:19:40 -06:00
a03598dfe2
feat: Implement Cross Chunk Schema / RecordBatch merging at query time ( #783 )
...
* feat: feat: Implement Cross Chunk Schema / RecordBatch merging at query time
* docs: update comments about NullArray::new_with-type
* docs: Update comments based on code review
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 18:26:38 +00:00
bc49fee920
refactor: Remove unneeded StringSetPlan::KnownError variant ( #775 )
...
* refactor: remove KnownError variant
* refactor: rename Known --> KnownOk
2021-02-10 11:53:10 -05:00
a316b16960
feat: Change table_names to return either Some(set) or None, rather than a plan (try 2) ( #776 )
...
* feat: Change table_names to return either Some(set) or None, rather than a plan
* docs: improve comments
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: merge conflict
* fix: don't clone a string unless needed
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-09 12:20:59 -05:00
74b8e8ec7d
refactor: Break Database query tests into their own files + add basic SQL tests ( #760 )
...
* refactor: break query tests into separate modules
* test: Add basic sql tests
2021-02-05 16:04:41 -05:00
b32e86e240
refactor: Remove async locking from test database ( #756 )
...
* refactor: Remove async locking from test database
* refactor: update service tests with test database
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-05 15:44:15 -05:00
47bc28460e
refactor: rename partition, table, and column in parition_meta for clarity ( #757 )
...
* refactor: rename partition, table, and column in parition_meta for clarity
2021-02-05 08:00:22 -05:00
fbf776c6b3
chore: Clean up Cargo.tomls ( #754 )
...
* fix: test_helpers crate should only be a dev-dep
* fix: object_store no longer has a build script, so no longer needs a build dep
* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
13b097728f
fix: Improve code, comments and error messages for consistency ( #737 )
...
* fix: Improve code, comments and error messages for consistency
* fix: update error message
2021-02-04 12:05:24 -05:00
202a4187a1
feat: Implement list databases API
...
Ref #722
2021-02-04 16:50:40 +01:00
d5ebf9c3da
chore: Update deps again ( #738 )
2021-02-04 06:02:05 -05:00
d66eae1a44
feat: Implement TableProvider for Trait for `Db` ( #730 )
...
* feat: Implement TableProvider for Db
Gets us selection pushdown in plans, sets us up for predicate pushdown
Includes: SendableRecordBatchStreams for mutable buffer and read buffer results
fixup snapshots
* docs: comments
2021-02-03 14:18:47 -05:00
abc26a33c1
chore: Update dependencies (again) ( #718 )
...
* chore: Update dependencies (again)
* refactor: update for changes in DataFusion API
* fix: fmt
* fix: clippy
2021-02-02 18:33:01 -05:00
288861e646
feat: implement table_schema in partition chunk, mutable buffer, read buffer ( #705 )
...
fix: sort output schema by name
fix: Update data_types/src/schema.rs
Co-authored-by: Edd Robinson <me@edd.io>
refactor: Update read_buffer/src/lib.rs
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2021-02-01 13:54:58 -05:00
f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) ( #707 )
...
* chore: Update arrow + tokio deps
* chore: Use bleeding edge azure
* chore: Update aws + other deps
* fix: fmt
* fix: Switch to in-house version of routerify
* fix: Upgrade to hyper 0.14
The hyper::error module is now private; hyper::Error is the public
re-export
* fix: Upgrade cloud storage to get tokio upgrade
* fix: Upgrade open_telemetry
* fix: Do not call `panic::set_hook` during another panic
Doing so leads to a double panic which aborts the process.
* fix: new h2 error who dis
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
2282a68e65
refactor: Move selection to the data_types crate and remove redundant implemenation ( #704 )
2021-01-29 13:35:07 -05:00
efb1e0f8ae
feat: Add selection interface to mutable buffer and query interface ( #700 )
...
* feat: Add selection interface to mutable buffer and query interface
* docs: Update mutable_buffer/src/table.rs
* refactor: rename for consistency
* refactor: use map and filter_map rather than fold
2021-01-27 14:31:10 -05:00
504ca67532
test: revamp rpc query testing so it works in multiple chunk scenarios ( #696 )
...
* test: revamp testing so it works in multiple scenarios, fix bug found by same
* fix: Update docs in server/src/db.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: use tsp rather than different functions
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-01-25 16:34:19 -05:00
c3b0371c84
feat: Initial RPC Query Frontend ( #692 )
...
* feat: Initial RPC Query Frontend
* docs: s/immutable buffer/mutable buffer
* docs: Correct type in docstring
2021-01-25 08:33:39 -05:00
7cb512ef67
refactor: Put tests in same module that defines struct ( #685 )
2021-01-21 19:33:15 -05:00
75b0a62fa5
refactor: Delete remove dead code ( #686 )
2021-01-21 19:20:39 -05:00
747b96d801
chore: Upgrade arrow dependencies, reduce duplication with upstream ( #676 )
2021-01-21 08:58:11 -05:00
6bb91653c1
refactor: Some tiny cleanups ( #680 )
...
* refactor: Remove import of unimplemented macro that's in the prelude
* refactor: Remove allowing of dead code that isn't dead anymore
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-01-21 07:27:17 -05:00
7969808f09
feat: Chunk Migration APIs and query data in the read buffer via SQL ( #668 )
...
* feat: Chunk Migration APIs and query data in the read buffer via SQL
* fix: Make code more consistent
* fix: fmt / clippy
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: Remove unecessary Result and make chunks() infallable
* chore: Apply more suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Edd Robinson <me@edd.io>
2021-01-19 13:28:26 -05:00
99605b27d7
chore: rename ( #660 )
2021-01-14 12:49:03 -05:00
8c0ce0b2ce
docs: Add query README file and explain some rationale ( #648 )
...
* docs: Add query README file
* fix: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2021-01-12 18:26:32 -05:00
a5240af080
docs: Document desired crate dependencies in comments ( #638 )
...
* docs: Document the desire for read buffer and mutable buffer to be independent of query layer
* docs: Document desire for the query layer to not depend on storage systems
* fix: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
2021-01-12 17:49:03 -05:00
6376891da3
feat: implement query planning in terms of chunks ( #647 )
2021-01-12 16:04:45 -05:00
c1a7778d85
refactor: move id and deps out of query crate ( #646 )
2021-01-12 11:47:43 -05:00
fd28d8a01b
refactor: Use u32 for Chunk ids consistently ( #639 )
2021-01-11 16:07:22 -05:00
a4be6f74c7
refactor: Remove partition key from the Chunk trait ( #622 )
2021-01-08 06:11:07 -05:00
c672bb341d
feat: Extract SQL planning out of databases ( #618 )
2021-01-07 13:13:30 -05:00
654b520005
feat: Interface for writing and querying mutable buffer, read buffer and parquet ( #615 )
...
* refactor: Create database with mutable buffer, read buffer and parquet files
* docs: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: rename planners to clarify what they are
* refactor: simplify traits
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-01-06 17:25:46 -05:00
08d52ea043
feat: implement partition chunk rollover + ids and timestamps ( #601 )
...
* feat: implement partition chunk rollover + ids and timestamps
* feat: add last_write_timestamp
* refactor: Use DateTime<Utc> rather than Instant
* refactor: avoid use of structure to generate ids
2020-12-29 11:00:18 -05:00
5fa77c32cc
feat: Add "Chunks" to the Mutable Buffer ( #596 )
...
* refactor: Update docs, remove unused field
* refactor: rename partition -> chunk
* feat: Introduce new partition, which is a holder for Chunks
* refactor: Remove use of wal from mutable database
* refactor: cleanups, remove last direct use of chunks
* fix: delete old benchmarks
* fix: clippy sacrifice
* docs: tidy up comments
* refactor: remove unused error types
* chore: remove commented out tests
2020-12-28 07:10:25 -05:00
48c43b136c
refactor: rename write_buffer --> mutable_buffer ( #595 )
...
* refactor: git mv write_buffer mutable_buffer
* refactor: update crate name references
* refactor: update some more references
2020-12-22 10:49:53 -05:00
263af1eeac
feat: implement read_group in the write_buffer ( #583 )
...
* feat: implement read_group in the write_buffer
* fix: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* fix: rustfmt
* fix: adjust tests for min/max
* fix: Update write_buffer/src/table.rs
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2020-12-22 09:03:20 -05:00
28eac06d8f
refactor: Organize window_bounds the same as selector functions ( #594 )
...
* refactor: Organize window_bounds the same as selector functions
* fix: add missing file
2020-12-21 12:51:36 -05:00
bb96142564
chore: Update arrow dependencies, remove custom min/max implementation ( #585 )
...
* chore: Update arrow dependency
* fix: Update code for changes in datafusion
* fix: use arrow version of min_boolean
2020-12-21 12:31:39 -05:00
0de5a1e309
feat: Implement selector functions first, last, min, and max ( #565 )
...
* feat: Implement selector functions first, last, min, and max
* fix: update for changes in arrow
* docs: reference to min/max boolean array ticket
* docs: add reference to selector structs
* docs: Update query/src/func/selectors.rs
2020-12-17 17:51:36 -05:00
a6d2c13888
chore: Update arrow + other depenencies ( #540 )
...
* chore: Update arrow + other depenencies
* chore: Update write_buffer and query crate
2020-12-15 08:46:27 -05:00
1740e26ec3
fix: do not produce gRPC series frames for fields that only contain null values ( #558 )
...
* test: add test for field columns with only nulls
* fix: do not produce series for null fields, tests for same
* fix: remove uneeded test printlns
2020-12-15 08:28:23 -05:00
4c35253fd5
style: unmangle wrapped diagrams
...
Adds #[rustfmt::skip] to comment blocks containing diagrams to skip wrapping.
2020-12-14 13:14:36 +00:00
6f473984d0
style: wrap comments
...
Runs rustfmt with the new config.
2020-12-11 18:22:26 +00:00
50ba529cb8
test: Adds tests for read_group for None aggregates ( #538 )
2020-12-10 11:31:15 -05:00
fa3ecbd4ed
feat: Implement write buffer to Parquet snapshotting ( #526 )
...
* feat: Implement write buffer to Parquet snapshotting
This introduces snapshot to the server packages to manage snapshotting. It also introduces a new trait for representing a Partition. There is a very crude API wired up in http_routes for testing purposes. Follow on work will bring the server package into http_routes and rework the snapshot API.
2020-12-08 14:20:43 -05:00
ceea61a211
Merge branch 'main' into dom/org-bucket-types
2020-12-04 11:33:36 +00:00
Carol (Nichols || Goulding)
Nga Tran
Marco Neumann
kodiakhq[bot]
Andrew Lamb
Jacob Marble
Raphael Taylor-Davies
Edd Robinson
Andrew Lamb
Carol (Nichols || Goulding)
Paul Dix
Jacob Marble
NGA TRAN
Jake Goulding
NGA TRAN
Paul Dix
Marko Mikulicic
Hu Ming
Dom