Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
9,866 Commits
203 Branches
453 Tags
381 MiB
Commit Graph

189 Commits (651b7a1ce64df19c730e1fd721a1dcf4ecf618cf)

Author SHA1 Message Date
Andrew Lamb 889804e46a
feat: avoid copying schemas while simplifying exprs (#3620) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-10 15:07:00 +00:00
Andrew Lamb d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively (#3633) 
* chore: update datafusion

* fix: Update to use new datafusion api

* chore: update expected plans

* fix: support zero output partitions

* fix: update test

* fix: Update for new DataFusion API

* fix: newly added system table

* fix: update cargo lock
 2022-02-09 19:53:41 +00:00
dependabot[bot] 803296e8cf
chore(deps): bump sqlparser from 0.13.0 to 0.14.0 (#3696) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.13.0 to 0.14.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.13.0...v0.14.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
 2022-02-09 19:40:03 +00:00
Carol (Nichols || Goulding) 2e30483f1f
refactor: Remove predicate module from predicate crate (#3648) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-07 14:54:07 +00:00
Andrew Lamb 77b80e7618
fix(InfluxQL): treat null tags as `''` rather than `null` in storagerpc queries (#3557) 
* fix(InfluxQL): treat null tags as `''` rather than `null` in storage rpc queries

* test: add one more case

* fix: Update comment

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>

Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-03 12:14:43 +00:00
Andrew Lamb 429d59f1b6
feat: Simplify predicates in the `InfluxRpcFrontend` before using them (#3588) 
* feat: normalize + simplify RPC predicates before using them

* docs: Update predicate/src/rpc_predicate.rs

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-02 19:46:57 +00:00
Andrew Lamb 030a2cb4c1
chore: Update datafusion (#3613) 
* chore: Update datafusion

* fix: update for latest DF API

* fix: another API change

* fix: clippy

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-02-02 16:27:11 +00:00
Edd Robinson c1f5994660
refactor: move sql parsing -> RPC predicate into own crate (#3604) 
* chore: create crate

* refactor: move module to new crate
 2022-02-02 10:41:57 +00:00
Andrew Lamb 0570662da2 feat: Fold past CASE blocks / translate to boolean CASE 2022-02-01 12:24:33 -05:00
Edd Robinson 1f0f6dad45 refactor: address PR feedback 2022-02-01 13:54:43 +00:00
Edd Robinson 506363dd1c refactor: expose API to generate RPC predicate 2022-02-01 12:47:37 +00:00
Edd Robinson 0b9315bfaa test: check _measurement and _field 2022-02-01 12:21:35 +00:00
Edd Robinson 4db2865558 feat: add support for precedence 2022-02-01 12:14:20 +00:00
Edd Robinson 2f4c559a5a feat: add support for ::field and ::tag 2022-02-01 11:06:51 +00:00
Edd Robinson 175732c3ca feat: basic sqlparser -> RPCNode 2022-02-01 10:26:03 +00:00
Raphael Taylor-Davies 21c1824a7a
refactor: remove table_names from Predicate (#3545) 
* refactor: remove table_names from Predicate

* chore: fix benchmarks

* chore: review feedback

Co-authored-by: Edd Robinson <me@edd.io>

* chore: review feedback

* chore: replace Default::default with InfluxRpcPredicate::default()

Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-27 14:44:49 +00:00
Andrew Lamb 5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 (#3517) 
* chore: Update datafusion

* chore: update to arrow 8

* fix: update to use new DataFusion APIs

* fix: update case for sortedness

* fix: cargo hakari
 2022-01-27 13:33:27 +00:00
Andrew Lamb 9615feacb3
fix(InfluxQL): Support RegEx with escape sequences not supported by Rust regex (#3502) 
* fix(InfluxQL): Translate unsupported meta characters

* fix: remove debugging

* fix: clippy sacrifice

* docs: Add additional background and rationale for rewriting

* fix: doc link
 2022-01-21 14:40:10 +00:00
Andrew Lamb 9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values (#3487) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-20 16:08:00 +00:00
Andrew Lamb f0d50f447a
fix: Special case tag_keys with max timestamp range (#3485) 
* fix: Special case tag_keys with max timestamp range

* docs: comment

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-20 14:14:34 +00:00
Andrew Lamb dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc (#3455) 
* chore: update datafusion, arrow, prost, tonic, etc

* fix: update pprof as well

* chore: update hakari

* fix: update pbjson

* chore: update heappy

* fix: hakari

* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2022-01-13 17:07:15 +00:00
Marco Neumann f3f6f335a9
chore: upgrade to snafu 0.7 (#3440) 2022-01-11 19:22:36 +00:00
Andrew Lamb ccba68fe3e
chore: Update datafusion and sqlparser, and arrow (#3385) 
* chore: Update datafusion and sqlparser, and arrow

* fix: cargo hakari

* test: fix metadata size

* fix: update some metadata sizes

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-12-16 12:13:40 +00:00
Carol (Nichols || Goulding) 02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost 2021-12-06 09:37:15 -05:00
Carol (Nichols || Goulding) 39862e3886
fix: Disable default chrono features 
Connects to #3117. Won't affect the workspace-hack crate until changes
to some other crates can be upstreamed.
 2021-12-06 09:36:49 -05:00
dependabot[bot] 5d784ba866
chore(deps): bump serde_json from 1.0.71 to 1.0.72 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.71 to 1.0.72.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.71...v1.0.72)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-29 09:28:07 +00:00
Marco Neumann a9ec0720b2 feat: allow to format delete predicates as SQL strings 
This is required to feed them back into the gRPC delete API.
 2021-11-23 15:39:53 +01:00
kodiakhq[bot] d16a7759ca
Merge branch 'main' into cn/workspace-hack 2021-11-22 17:05:31 +00:00
dependabot[bot] 5c6deb81db
chore(deps): bump serde_json from 1.0.70 to 1.0.71 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-22 01:21:06 +00:00
Carol (Nichols || Goulding) 9fd4a560f5
feat: Results of running cargo hakari manage-deps 2021-11-19 09:21:57 -05:00
Raphael Taylor-Davies 58f3e2e559
refactor: move delete predicate proto serialization to generated_types (#3108) 
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-15 12:02:14 +00:00
Raphael Taylor-Davies 3d091208af
refactor: move delete predicate into data_types (#2731) (#3094) 
* refactor: move delete predicate into dml (#2731)

* refactor: move DeletePredicate to data_types

* chore: fix doc

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-15 10:28:58 +00:00
dependabot[bot] 3485d70409
chore(deps): bump serde_json from 1.0.69 to 1.0.70 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-15 08:48:44 +00:00
Raphael Taylor-Davies 6320ce6f55
refactor: move delete predicate proto to own package (#2731) (#3065) 
* refactor: move delete predicate proto to own package (#2731)

* chore: fmt

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-11-08 13:42:26 +00:00
dependabot[bot] 7214fa9d6a
chore(deps): bump serde_json from 1.0.68 to 1.0.69 
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-08 10:31:33 +00:00
Marco Neumann 4c9570b519 refactor: move `catalog` protobuf to `preserved_catalog` 
This makes it clearer what's going since the contained messages are
only for the preserved part, not the in-mem catalog and its management.
 2021-11-01 18:07:25 +01:00
dependabot[bot] c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-11-01 11:21:59 +00:00
Andrew Lamb 7cd56cbc56 Merge remote-tracking branch 'origin/main' into er/fix/flux/2691 2021-10-25 13:41:08 -04:00
Marco Neumann bc7244c48e chore: use Rust edition 2021 2021-10-25 10:58:20 +02:00
Andrew Lamb 52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate (#2947) 
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate

* fix: make apply_predicate_to_metadata precise

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-24 11:03:30 +00:00
Edd Robinson 6683a1e629
Merge branch 'main' into er/fix/flux/2691 2021-10-22 16:28:21 +01:00
Edd Robinson 84a00cd5b6 refactor: comma 2021-10-22 15:35:24 +01:00
Edd Robinson 3d9f9ea53e refactor: use Expr Display impl 2021-10-21 22:43:13 +01:00
Edd Robinson 0cb9751b0a refactor: extract _value expr from predicate 2021-10-21 22:26:32 +01:00
Andrew Lamb a82dc6f5f0
chore: Update datafusion + arrow (#2903) 
* chore: Update datafusion to latest, arrow to 6.0.0

* fix: Update tests

* fix: bubble internal error

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-19 17:14:08 +00:00
dependabot[bot] c5405fbd5f
chore(deps): bump sqlparser from 0.11.0 to 0.12.0 (#2867) 
Bumps [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) from 0.11.0 to 0.12.0.
- [Release notes](https://github.com/sqlparser-rs/sqlparser-rs/releases)
- [Changelog](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/sqlparser-rs/sqlparser-rs/compare/v0.11.0...v0.12.0)

---
updated-dependencies:
- dependency-name: sqlparser
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-18 14:11:02 +00:00
Nga Tran fbf5539336 chore: merge main to branch 2021-10-11 10:47:10 -04:00
Raphael Taylor-Davies afe34751e7
refactor: split out schema crate (#2781) 
* refactor: split out schema crate

* chore: fix doc
 2021-10-11 09:45:08 +00:00
Nga Tran d0a17ca79d refactor: address Edd's review comments 2021-10-08 16:47:37 -04:00
Nga Tran f7475322a6 chore: merge main to branch, resolve conflicts, and discover an inconsitent bug 2021-10-08 15:50:46 -04:00
Nga Tran f2cdb9531f chore: cleanup 2021-10-08 14:52:15 -04:00
Nga Tran 22d6f11bea fix: add cols of delete predicates into the schema of scanning columns 2021-10-07 17:37:34 -04:00
Andrew Lamb c7727f1b5b
chore: Update datafusion + other deps (#2760) 
* chore: Update datafusion and other deps

* fix: fmt

* fix: cleanup workaround

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-10-07 16:48:11 +00:00
kodiakhq[bot] d72a494198
Merge branch 'main' into crepererum/in_mem_expr_part5 2021-10-05 16:20:24 +00:00
Nga Tran 825ddaa34c chore: merge main to branch 2021-10-05 10:12:17 -04:00
Marco Neumann bb7a27e5ed refactor: use proper sets during delete predicate collection 
We no longer need hacky pointer tricks to de-duplicate delete predicates
when collecting them for catalog checkpoints. This was once required
when the delete predicates didn't implement `Eq` and `Hash` but now it's
all way easier.
 2021-10-05 10:37:34 +02:00
Marco Neumann 1d18758075 refactor: fold `ParseDeletePredicate` into `DeletePredicate` 
There is no reason to have yet another intermediate type. This also
makes the intermediate parsing functions private.
 2021-10-05 09:29:25 +02:00
Marco Neumann 10c1a72402 refactor: remove unused fields from `DeletePredicate` 2021-10-05 09:29:24 +02:00
Nga Tran ff8e037971 feat: handle delete response 2021-10-04 20:37:53 -04:00
Nga Tran 1856d7184c fix: make stop time inclusive 2021-10-04 14:14:16 -04:00
Marco Neumann 150e7c222f chore: enable linting for `predicate` crate 2021-10-04 17:58:40 +02:00
Marco Neumann d8dddc358f
fix: typo in error message 
Co-authored-by: Nga Tran <ntran@influxdata.com>
 2021-10-04 16:56:43 +02:00
Marco Neumann 75ac6e8646 refactor: make `DeletePredicate::range` non-optional 2021-10-04 16:36:20 +02:00
Marco Neumann d1835a3eee fix: doc links 2021-10-04 16:36:20 +02:00
Marco Neumann 5a5a929b9e refactor: introduce `DeletePredicate` 
`DeletePredicate` is a simpler version of `Predicate` that is based on
IOx `DeleteExpr` instead of the full-blown DataFusion `Expr`. This will
allow us to do a couple of things (in follow-up changes):

- Order and de-duplicate delete predicates
- Normalize predicates
- Infallible serialization
- Smaller memory footprint

Note that this change only affects delete expressions. Query expressions
that are supported via the API are not changed. The query subsystem also
still uses the full-featured expressions/predicates (delete
expressions/predicates are converted to the more powerful DataFusion
version on-the-fly).
 2021-10-04 16:36:20 +02:00
dependabot[bot] d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0 
Bumps [arrow](https://github.com/apache/arrow-rs) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases)
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md)
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0)

---
updated-dependencies:
- dependency-name: arrow
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2021-10-04 08:55:38 +00:00
Marco Neumann 51246e3c8d docs: remove outdated comment 2021-10-04 10:44:30 +02:00
Marco Neumann ec0f256ad7 refactor: `Expr` => `DeleteExpr` 2021-10-04 10:28:09 +02:00
Marco Neumann a0bbbdb197 refactor: use dedicated in-memory `Expr` type during IO 
Place a dedicated `Expr` type between datafusion and protobuf. While
this type is currently only used during serialization, it will be used
within `Predicate` in a follow-up change to enable de-duplication of
predicates and avoid the double parsing of datafusion expressions (we
are already somewhat parsing them when we check for valid delete
predicates).

This change looks larger than it is. In practice it just separates the
conversion "datafusion => protobuf" into "datafusion => IOx => protobuf"
and "protobuf => datafusion" into "protobuf => IOx => datafusion". So
(apart from the error types) this is functionally the same.
 2021-10-04 09:59:14 +02:00
Nga Tran 154dd4460e refactor: address review comments 2021-10-01 16:37:22 -04:00
Nga Tran ee94e9038a test: finalize codin up delete http endpoints and end-to-end tests 2021-10-01 12:15:00 -04:00
Nga Tran 53a45d89b7 feat: support HTTP Delete Endpoints 2021-09-30 18:09:17 -04:00
Andrew Lamb a55a21c644
chore: Update datafusion (#2635) 
* chore: Update datafusion and sqlparser

* fix: remove STACK_SIZE workaround

* chore: update datafusion_util

* chore: update predicate

* chore: update query_tests

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
 2021-09-27 14:13:19 +00:00
Nga Tran e80d8aa391 chore: update latest datafusion and use its lit_timestamp_nano 2021-09-22 11:30:42 -04:00
Nga Tran 2f371b6f79 refactor: address review comments 2021-09-21 14:46:24 -04:00
Nga Tran 85989cc8a3 test: add more delete tests and test scenarios 2021-09-20 18:18:08 -04:00
kodiakhq[bot] c7e6fffaaa
Merge branch 'main' into ntran/delete_scan 2021-09-20 13:29:47 +00:00
Marco Neumann acf698c366 fix: delete predicate sorting 2021-09-20 10:48:32 +02:00
Nga Tran 364d245eae feat: apply negated delete predicates during scan 2021-09-17 16:20:42 -04:00
Nga Tran 60a866ddcb refactor: merge delete predicates into select predicate 2021-09-17 07:52:33 -04:00
Marco Neumann ec943081c7 refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates 
The motivations are:

1. The API uses a SINGLE predicate and adds that to many chunks. With
   `Arc<Vec<...>>` you gain nothing, with `Vec<Arc<...>>` the predicate
   is only stored once (in many vectors)
2. While we currently add predicates blindly to all chunks, we can be way
   smarter in the future and prune out tables, partitions or even single
   chunks (based on statistics). With that, it will be rare that many
   chunks share the exact same set of predicates.
3. It would be nice if we could de-duplicate predicates when writing them
   to the preserved catalog without needing to repeat the pruning
   discussed in point 2. This is way easier to implement whan chunks
   exists in `Arc`s.
4. As a side-note: the `Arc<Vec<...>>` wasn't really cloned around but
   instead was created many time. So the new version should be more
   memory efficient out of the box.
 2021-09-16 17:16:09 +02:00
kodiakhq[bot] 33cd1cffad
Merge branch 'main' into ntran/delete_read 2021-09-16 13:22:50 +00:00
Nga Tran 61e1eac135 fix: fix the cases of multi[le expressions in delete predicate 2021-09-15 17:00:21 -04:00
Marco Neumann 72021e010f test: test unsupported cases of predicate serialization 2021-09-15 18:18:05 +02:00
Marco Neumann c5ebf4a2e6 refactor: remove unused predicate serialization variants 2021-09-15 18:05:11 +02:00
Marco Neumann 44eb3b994d feat: `Predicate` serialization 
Closes #2493.
 2021-09-15 16:37:25 +02:00
Nga Tran 63cc7b3fb0 test: more tests to discover what still need to be done 2021-09-14 17:57:30 -04:00
Nga Tran f4f140d3b7 chore: merge main to branch 2021-09-14 13:25:32 -04:00
Marco Neumann bfaba78dc3 refactor: move `predicate` into its own crate 
Two reasons:

1. I wanna decouple `parquet_file` from `query` (nearly done, needs a
   small follow-up PR).
2. `predicate` will have more and more features (like serialization)
   which justifies a new home
 2021-09-14 17:13:02 +02:00