Commit Graph

85 Commits (488252d9f106497f1805a2e2638f1df5c6b39e05)

Author SHA1 Message Date
Andrew Lamb 746373a687
refactor: Remove mutable_buffer crate dependency on query crate (#927) 2021-03-05 11:34:27 +00:00
Andrew Lamb 8b1f100df3
feat: make read_group and read_window_aggregate work across chunks (#905)
* feat: make read_group and read_window_aggregate work across chunks

* refactor: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: Update query/src/frontend/influxrpc.rs

Improve logic and use strings directly

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-04 17:06:31 +00:00
Nga Tran 957e05ef25 chore: use newly added Arrow's Expr::is_not_null function 2021-03-03 11:46:49 -05:00
Andrew Lamb 94bd200e60
refactor: Add Predicate::is_empty() and EMPTY_PREDICATE to avoid unecessary construction (#891)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-03-01 21:03:05 +00:00
Andrew Lamb 7d8d00781c
feat: Make read_filter work for mutable buffer and read buffer (#882)
* feat: port read_filter to InfluxRPCPlanner

* fix: remove commented out vestigal test

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: fmt

* fix: Update arrow_deps/src/util.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-03-01 16:50:29 +00:00
Nga Tran 6ad8e1aa33 feat: use newly implemented tags_iter to get Tag columns 2021-02-26 15:54:20 -05:00
Nga Tran 18de3bdcab chore: merge main into branch
Merge branch 'main' into ntran/optimize_column_selection
2021-02-26 15:29:43 -05:00
Nga Tran f37e5846aa feat: fmt auto fix 2021-02-26 14:56:10 -05:00
NGA TRAN eb81975151 feat: Optimize Column Selection 2021-02-26 14:28:46 -05:00
Andrew Lamb 12deacd8a0
refactor: move SeriesSetPlans into its own module (#878)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-25 23:12:39 +00:00
Andrew Lamb 8fb7651719
feat: Port tag_values to the InfluxRPCPlanner (#859)
* feat: Port tag_values to the InfluxRPCPlanner

* refactor: merge imports

* refactor: rename column_names to tag_column_names for clarity

* fix: Update query/src/frontend/influxrpc.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: use ensure!

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: less silly whitespace

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: code review comments

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-24 23:11:22 +00:00
Andrew Lamb ed7859e182
refactor: rename tag_column_names --> tag_keys in InfluxRPCPlanner (#860) 2021-02-23 17:04:53 +00:00
Carol (Nichols || Goulding) 0aee605544 Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction 2021-02-22 14:13:34 -05:00
Jake Goulding 6e6cc616a0 refactor: Switch to parking_lot::Mutex 2021-02-22 13:51:31 -05:00
Edd Robinson 4caa1d732f refactor: make Selection Copy 2021-02-22 15:32:55 +00:00
Carol (Nichols || Goulding) d0707725cf Merge remote-tracking branch 'origin/main' into pd-mutable-buffer-data-eviction 2021-02-22 10:21:59 -05:00
Edd Robinson 92eb8b9e85 refactor: make certain Database method sync
A couple of methods don't seem to have any await points in their
implementations, so it feels like they could just be `sync`.
2021-02-19 17:14:17 +00:00
kodiakhq[bot] 4718b340f3
Merge branch 'main' into ntran/update_arrow_dependencies 2021-02-18 16:20:49 +00:00
NGA TRAN 213094f8f7 chore: update Arrow dependencies 2021-02-18 10:02:57 -05:00
Andrew Lamb 9b91e0624c
feat: implement field_columns plan (#819)
* feat: implement field_columns plan

* fix: fix doc tests

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-17 20:43:24 +00:00
Andrew Lamb 071b13b939
chore: Update dependencies (#821)
* chore: Update dependencies

* fix: update udf implementation for DataFusion update

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-16 23:27:36 +00:00
Andrew Lamb 94a93e56ff
feat: implement `tag_keys` in gRPC planner and across mutable buffer (#795)
* feat: move tag_column_names into rpc planner

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: compile error

* refactor: remove PassThrough error type

* fix: Avoid extra layers of errors in mutable buffer chunk

* fix: use HashMap::get rather than values() and find

* fix: push filtering down to chunk in gRPC planner

* fix: fixup trait bounds to be non-silly

* fix: remove incorrect comment

* fix: remove cruft

* fix: clippy + fmt

* fix: correct comment

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-15 16:47:52 +00:00
Edd Robinson 8eaf006c27 refactor: add arc clone lint 2021-02-15 12:35:21 +00:00
Andrew Lamb 11c97a6235 fix: remove make_null_columns 2021-02-14 05:51:25 -05:00
Paul Dix dc465e5d02 feat: Add function to check db size and drop partitions
Adds functionality to the server Db to check the mutable buffer size and drop partitions based on the database rules.
2021-02-13 17:19:40 -06:00
Paul Dix 83bfa6d949 feat: Add created_at, last_write_at tracking to partition and sorting
This commit adds created_at and last_write_at instants to partitions in the mutable buffer. It adds a method on the mutable buffer database to get back the partitions in sorted order based on either the created_at or last_write_at instants. Ordering based on the summary stats from a column are still left to do.

Finally, it modifies the helper function to create replicated write to take a Partitioner trait that can generate partition keys based on lines, rather than taking the DatabaseRules struct directly. This makes it easier to write test cases where data is split into multiple partitions in the mutable buffer.
2021-02-13 17:19:40 -06:00
Andrew Lamb a03598dfe2
feat: Implement Cross Chunk Schema / RecordBatch merging at query time (#783)
* feat: feat: Implement Cross Chunk Schema / RecordBatch merging at query time

* docs: update comments about NullArray::new_with-type

* docs: Update comments based on code review

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-02-11 18:26:38 +00:00
Andrew Lamb bc49fee920
refactor: Remove unneeded StringSetPlan::KnownError variant (#775)
* refactor: remove KnownError variant

* refactor: rename Known --> KnownOk
2021-02-10 11:53:10 -05:00
Andrew Lamb a316b16960
feat: Change table_names to return either Some(set) or None, rather than a plan (try 2) (#776)
* feat: Change table_names to return either Some(set) or None, rather than a plan

* docs: improve comments

* docs: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* fix: merge conflict

* fix: don't clone a string unless needed

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-09 12:20:59 -05:00
Andrew Lamb 74b8e8ec7d
refactor: Break Database query tests into their own files + add basic SQL tests (#760)
* refactor: break query tests into separate modules

* test: Add basic sql tests
2021-02-05 16:04:41 -05:00
Andrew Lamb b32e86e240
refactor: Remove async locking from test database (#756)
* refactor: Remove async locking from test database

* refactor: update service tests with test database

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-02-05 15:44:15 -05:00
Paul Dix 47bc28460e
refactor: rename partition, table, and column in parition_meta for clarity (#757)
* refactor: rename partition, table, and column in parition_meta for clarity
2021-02-05 08:00:22 -05:00
Carol (Nichols || Goulding) fbf776c6b3
chore: Clean up Cargo.tomls (#754)
* fix: test_helpers crate should only be a dev-dep

* fix: object_store no longer has a build script, so no longer needs a build dep

* chore: Alphabetize all Cargo.tomls
2021-02-04 18:56:02 -05:00
Andrew Lamb 13b097728f
fix: Improve code, comments and error messages for consistency (#737)
* fix: Improve code, comments and error messages for consistency

* fix: update error message
2021-02-04 12:05:24 -05:00
Marko Mikulicic 202a4187a1
feat: Implement list databases API
Ref #722
2021-02-04 16:50:40 +01:00
Andrew Lamb d5ebf9c3da
chore: Update deps again (#738) 2021-02-04 06:02:05 -05:00
Andrew Lamb d66eae1a44
feat: Implement TableProvider for Trait for `Db` (#730)
* feat: Implement TableProvider for Db

Gets us selection pushdown in plans, sets us up for predicate pushdown

Includes: SendableRecordBatchStreams for mutable buffer and read buffer results

fixup snapshots

* docs: comments
2021-02-03 14:18:47 -05:00
Andrew Lamb abc26a33c1
chore: Update dependencies (again) (#718)
* chore: Update dependencies (again)

* refactor: update for changes in DataFusion API

* fix: fmt

* fix: clippy
2021-02-02 18:33:01 -05:00
Andrew Lamb 288861e646
feat: implement table_schema in partition chunk, mutable buffer, read buffer (#705)
fix: sort output schema by name

fix: Update data_types/src/schema.rs

Co-authored-by: Edd Robinson <me@edd.io>

refactor: Update read_buffer/src/lib.rs

Co-authored-by: Edd Robinson <me@edd.io>

Co-authored-by: Edd Robinson <me@edd.io>
2021-02-01 13:54:58 -05:00
Andrew Lamb f3bd8bd0e3
chore: update deps (tokio 1.0 and ecosystem) (#707)
* chore: Update arrow + tokio deps

* chore: Use bleeding edge azure

* chore: Update aws + other deps

* fix: fmt

* fix: Switch to in-house version of routerify

* fix: Upgrade to hyper 0.14

The hyper::error module is now private; hyper::Error is the public
re-export

* fix: Upgrade cloud storage to get tokio upgrade

* fix: Upgrade open_telemetry

* fix: Do not call `panic::set_hook` during another panic

Doing so leads to a double panic which aborts the process.

* fix: new h2 error who dis

Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@integer32.com>
Co-authored-by: Jake Goulding <jake.goulding@integer32.com>
2021-01-29 16:11:55 -05:00
Andrew Lamb 2282a68e65
refactor: Move selection to the data_types crate and remove redundant implemenation (#704) 2021-01-29 13:35:07 -05:00
Andrew Lamb efb1e0f8ae
feat: Add selection interface to mutable buffer and query interface (#700)
* feat: Add selection interface to mutable buffer and query interface

* docs: Update mutable_buffer/src/table.rs

* refactor: rename for consistency

* refactor: use map and filter_map  rather than fold
2021-01-27 14:31:10 -05:00
Andrew Lamb 504ca67532
test: revamp rpc query testing so it works in multiple chunk scenarios (#696)
* test: revamp testing so it works in multiple scenarios, fix bug found by same

* fix: Update docs in server/src/db.rs

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: use tsp rather than different functions

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2021-01-25 16:34:19 -05:00
Andrew Lamb c3b0371c84
feat: Initial RPC Query Frontend (#692)
* feat: Initial RPC Query Frontend

* docs: s/immutable buffer/mutable buffer

* docs: Correct type in docstring
2021-01-25 08:33:39 -05:00
Andrew Lamb 7cb512ef67
refactor: Put tests in same module that defines struct (#685) 2021-01-21 19:33:15 -05:00
Andrew Lamb 75b0a62fa5
refactor: Delete remove dead code (#686) 2021-01-21 19:20:39 -05:00
Andrew Lamb 747b96d801
chore: Upgrade arrow dependencies, reduce duplication with upstream (#676) 2021-01-21 08:58:11 -05:00
Carol (Nichols || Goulding) 6bb91653c1
refactor: Some tiny cleanups (#680)
* refactor: Remove import of unimplemented macro that's in the prelude

* refactor: Remove allowing of dead code that isn't dead anymore

Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-01-21 07:27:17 -05:00
Andrew Lamb 7969808f09
feat: Chunk Migration APIs and query data in the read buffer via SQL (#668)
* feat: Chunk Migration APIs and query data in the read buffer via SQL

* fix: Make code more consistent

* fix: fmt / clippy

* chore: Apply suggestions from code review

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

* refactor: Remove unecessary Result and make chunks() infallable

* chore: Apply more suggestions from code review

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Edd Robinson <me@edd.io>
2021-01-19 13:28:26 -05:00
Hu Ming 99605b27d7
chore: rename (#660) 2021-01-14 12:49:03 -05:00