3ab0788a94
fix: Move DeletePredicate types to data_types2
2022-05-06 14:45:37 -04:00
485d6edb8f
refactor: Move IngesterQueryRequest to generated_types
2022-05-06 14:45:37 -04:00
96f0c88b48
fix: Remove db crate from query_tests
2022-05-06 11:30:36 -04:00
7286b4391a
fix: Remove db crate
2022-05-06 11:27:33 -04:00
02893e598c
chore: Update datafusion and upgrade arrow/parquet/arrow-flight to 13 ( #4516 )
...
* chore: Tool for automating arrow version update
* chore: Update datafusion and arrow/parquet/arrow-flight
* fix: update for changes in Arrow API
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-05 00:21:02 +00:00
e015d3bafb
feat: Remove the server_benchmarks crate ( #4506 )
2022-05-02 18:20:01 +00:00
799480d34e
refactor: query_tests - port a few OG to NG tests and remove many more that already ported ( #4487 )
...
* refactor: port a few OG to NG tests and remove many more that already ported
* chore: Apply suggestions from code review
* chore: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-05-02 14:44:02 +00:00
6eed09a926
test: use "real" ingester in query tests ( #4455 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-28 14:39:31 +00:00
420c306caa
chore(deps): Bump tokio from 1.17.0 to 1.18.0 ( #4453 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.17.0...tokio-1.18.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-28 08:21:17 +00:00
9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) ( #4428 )
...
* refactor: Move all UDF implementations to query_function crate
* refactor: Move regex udf to query_functions
* refactor: Move functions out of query
* fix: lints, imports
* chore: Run cargo hakari tasks
* fix: clipy + benches
* fix: reduce borrowing and fix clippy
* fix: moar clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 17:30:27 +00:00
2337935660
test: chunks in ingester stage ( #4415 )
...
* refactor: document and improve `MockIngesterConnection`
* refactor: split `OldOneMeasurementFourChunksWithDuplicates` for `EXPLAIN` queries
* fix: mark "IngsterPartition" chunks as unsorted
* fix: "group by" queries may require sorted comparison
* refactor: re-export a few more types from querier
* fix: ensure that test parquet files are de-duped
* test: chunks in ingester stage
* docs: explain test code
2022-04-26 07:55:19 +00:00
4c94e46642
chore(deps): Bump croaring from 0.5.2 to 0.6.0 ( #4408 )
...
* chore(deps): Bump croaring from 0.5.2 to 0.6.0
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.5.2 to 0.6.0.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.5.2...0.6.0 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: croaring 0.6.0 compat
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-04-25 16:41:08 +00:00
f444e63960
test: include materialized delete predicates in NG query tests ( #4371 )
...
* refactor: move `batch_filter` to `datafusion_util`
* fix: outdated docstring
* feat: allow passing record batches to `iox_tests` parquet files
* test: include materialized delete predicates in NG query tests
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 13:00:13 +00:00
73bed810da
chore: Update arrow, arrow-flight, parquet, tonic, prost, etc ( #4357 )
...
* chore: Update datafusion
* chore: Update arrow/arrow-flight/parquet to 12
* chore: update datafusion correctly
* chore: Update prost, tonic, and dependents
* fix: Fixup some api changes
* fix: Update test output in db
* fix: Update test output in parquet_file
* fix: remove old pbjson types
* fix: Add "--experimental_allow_proto3_optional" flag
* chore: Run cargo hakari tasks
* fix: compile error
* chore: Update heappy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 11:12:17 +00:00
f6e6821276
feat: Add basic Querier <--> Ingester "Service Configuration" ( #4259 )
...
* feat: Add basic Querier <--> Ingester "Service Configuration"
* docs: update comments in test
* refactor: cleanup tests a little
* refactor: make trait more consistent
* docs: improve comments in IngesterPartition
2022-04-11 11:50:22 +00:00
bbbdcc75a8
feat: `QuerierDatabase::chunks` returns `Result` ( #4260 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 18:54:17 +00:00
b1af5b3f44
feat: query log system table for querier ( #4157 )
...
* feat: query log system table for querier
Closes #4084 .
* fix: typo
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* docs: extend
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-30 15:38:11 +00:00
2b76c31157
refactor: make statistics null counts optional ( #4160 )
...
Min/max values and distinct counts are already optional, so let's make
the null counts optional as well. This will be helpful for NG to deal w/
partial statistics (e.g. we only populate stats for the time column).
Note that the total count is still mandatory, but we normally have the
chunk/file-level row count at hand.
2022-03-29 17:47:57 +00:00
7d947c79d5
refactor: small query tests clean up ( #4156 )
...
* refactor: make NG query test generation more flexible
* refactor: rename OG-specfic query tests
* docs: explain chunk stage generation in NG query tests
* fix: typo
2022-03-29 14:00:34 +00:00
decd018a6a
refactor: remove querier sync loop ( #4146 )
...
Namespaces are now created on demand and contain their full schema.
Tombstones/chunks are created on demand during the query.
Closes #4123 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-29 11:47:23 +00:00
58c630d709
chore: Update datafusion ( #4133 )
...
* chore: Update datafusion
* fix: typo
* fix: Update explain plan output
* fix: update Cargo.locl
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-25 15:08:39 +00:00
9886ff42cc
refactor: clean up querier public interface
2022-03-25 11:54:52 +01:00
5c69a3f43b
chore: Update deps: datafusion, arrow/arrow-flight/parquet to 11, zstd to 0.11 ( #4119 )
...
* chore: update datafusion
* chore(deps): Bump arrow from 10.0.0 to 11.0.0
Bumps [arrow](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump arrow-flight from 10.0.0 to 11.0.0
Bumps [arrow-flight](https://github.com/apache/arrow-rs ) from 10.0.0 to 11.0.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/10.0.0...11.0.0 )
---
updated-dependencies:
- dependency-name: arrow-flight
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: update parquet to 11.0.0
* fix: error on create schema, test for same
* fix: upgrade zstd
* chore: Run cargo hakari tasks
* fix: fix logical merge conflict
* fix: hakari
* fix: hakari
* fix: update newly introduced dep
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:27:36 +00:00
8ca5c337b2
refactor: port more query tests to NG, some code clean up ( #4125 )
...
* refactor: inline function that is used once
* refactor: generalize multi-chunk creation for NG
* refactor: `TwoMeasurementsManyFieldsTwoChunks` is OG-specific
* refactor: generalize `OneMeasurementTwoChunksDifferentTagSet`
* refactor: port `OneMeasurementFourChunksWithDuplicates` to NG
* refactor: `TwoMeasurementsManyFieldsLifecycle` is OG-specific
* refactor: simplify NG chunk generation
* refactor: port `ThreeDeleteThreeChunks` to NG
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 15:07:09 +00:00
cc7f744e8e
test: two-chunk scenarios for NG ( #4113 )
...
Add the generic components to create two-chunk scenarios. Includes small
scenario fixes for things like system tables that are not identical
between OG and NG (also see #4111.)
Ref #3934 .
2022-03-24 09:50:57 +00:00
283d3dad5d
refactor: generalize query test scenarios a bit ( #4103 )
...
Some query test scenarios are duplicates and are very OG specific. Let's
use generic scenarios (i.e. the ones that contain all chunk stages
instead of a specific one) where applicable.
For #3934 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-24 09:30:19 +00:00
93485a11ec
Merge branch 'main' into crepererum/issue3934h
2022-03-23 19:10:02 +00:00
c33ef79375
test: improve query test runner output ( #4112 )
...
- prints more previous/expected values when failing (instead of just
emitting an `Err` which will be debug-printed)
- fixed newline handling (i.e. do not add additional newlines in
`PrintlnWriter::write`)
Before:
```text
Running scenario 'Two chunks: NG Chunk Parquet; NG Chunk Parquet'
SQL: '"SELECT * from information_schema.tables;"'
thread 'cases::test_cases_sql_information_schema_sql' panicked at 'test failed: ScenarioMismatch { scenario_name: "Two chunks: NG Chunk Parquet; NG Chunk Parquet", previous_results: ["+---------------+--------------------+---------------------+------------+", "| table_catalog | table_schema | table_name | table_type |", "+---------------+--------------------+---------------------+------------+", "| public | information_schema | columns | VIEW |", "| public | information_schema | tables | VIEW |", "| public | iox | h2o | BASE TABLE |", "| public | iox | o2 | BASE TABLE |", "| public | system | chunk_columns | BASE TABLE |", "| public | system | chunks | BASE TABLE |", "| public | system | columns | BASE TABLE |", "| public | system | operations | BASE TABLE |", "| public | system | persistence_windows | BASE TABLE |", "| public | system | queries | BASE TABLE |", "+---------------+--------------------+---------------------+------------+"], current_results: ["+---------------+--------------------+------------+------------+", "| table_catalog | table_schema | table_name | table_type |", "+---------------+--------------------+------------+------------+", "| public | information_schema | columns | VIEW |", "| public | information_schema | tables | VIEW |", "| public | iox | h2o | BASE TABLE |", "| public | iox | o2 | BASE TABLE |", "+---------------+--------------------+------------+------------+"] }', query_tests/src/cases.rs:169:10
stack backtrace:
0: rust_begin_unwind
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
```
After:
```text
Running scenario 'Two chunks: NG Chunk Parquet; NG Chunk Parquet'
SQL: '"SELECT * from information_schema.tables;"'
Answers produced by scenario Two chunks: NG Chunk Parquet; NG Chunk Parquet differ from previous answer
previous:
+---------------+--------------------+---------------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+---------------------+------------+
| public | information_schema | columns | VIEW |
| public | information_schema | tables | VIEW |
| public | iox | h2o | BASE TABLE |
| public | iox | o2 | BASE TABLE |
| public | system | chunk_columns | BASE TABLE |
| public | system | chunks | BASE TABLE |
| public | system | columns | BASE TABLE |
| public | system | operations | BASE TABLE |
| public | system | persistence_windows | BASE TABLE |
| public | system | queries | BASE TABLE |
+---------------+--------------------+---------------------+------------+
current:
+---------------+--------------------+------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+------------+------------+
| public | information_schema | columns | VIEW |
| public | information_schema | tables | VIEW |
| public | iox | h2o | BASE TABLE |
| public | iox | o2 | BASE TABLE |
+---------------+--------------------+------------+------------+
thread 'cases::test_cases_sql_information_schema_sql' panicked at 'test failed: ScenarioMismatch { scenario_name: "Two chunks: NG Chunk Parquet; NG Chunk Parquet", previous_results: ["+---------------+--------------------+---------------------+------------+", "| table_catalog | table_schema | table_name | table_type |", "+---------------+--------------------+---------------------+------------+", "| public | information_schema | columns | VIEW |", "| public | information_schema | tables | VIEW |", "| public | iox | h2o | BASE TABLE |", "| public | iox | o2 | BASE TABLE |", "| public | system | chunk_columns | BASE TABLE |", "| public | system | chunks | BASE TABLE |", "| public | system | columns | BASE TABLE |", "| public | system | operations | BASE TABLE |", "| public | system | persistence_windows | BASE TABLE |", "| public | system | queries | BASE TABLE |", "+---------------+--------------------+---------------------+------------+"], current_results: ["+---------------+--------------------+------------+------------+", "| table_catalog | table_schema | table_name | table_type |", "+---------------+--------------------+------------+------------+", "| public | information_schema | columns | VIEW |", "| public | information_schema | tables | VIEW |", "| public | iox | h2o | BASE TABLE |", "| public | iox | o2 | BASE TABLE |", "+---------------+--------------------+------------+------------+"] }', query_tests/src/cases.rs:169:10
stack backtrace:
0: rust_begin_unwind
at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
```
2022-03-23 18:06:09 +00:00
5ae1e2fecf
refactor: make query tests less OG-specific
2022-03-23 12:04:32 +01:00
89206e013c
test: run SOME query tests for querier ( #4098 )
...
This includes some type changes to dispatch between OG and NG and allows
some tests to be run against the NG querier. This only contains parquet
files though, so it's somewhat a limited scope.
For #3934 .
2022-03-22 17:39:19 +00:00
b83b000590
chore: Update datafusion ( #4071 )
...
* chore: update to datafusion 5936edc2a94d5fb20702a41eab2b80695961b9dc
* chore: Update apis to match datafusion changes
2022-03-22 13:17:41 +00:00
c9908b260c
refactor: dyn-dispatch database in query subsystem ( #4083 )
...
* refactor: dyn-dispatch database in query subsystem
This is similar to #4080 but concerns the database itself.
For #3934 .
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-22 09:15:52 +00:00
d1df95df87
refactor: dyn-dispatch chunks in query subsystem
...
- this is what DataFusion is doing as well; it's also fast enough
because the number of chunks in a query is not THAT massive (it's not
like we are doing row-level dyn dispatching)
- it simplifies abstracting over different databases
- it allows us to drop our enum-based dispatching that we have for
`DbChunk` and that we would also need for the querier (e.g. depending
on if a chunk is backed by a parquet file or ingester data)
- it likely speeds up compile times because the `query` is no longer
contains massive amounts of generic code
For #3934 .
2022-03-21 12:47:54 +01:00
ca152e7934
refactor: avoid generics in `QueryDatabase`
...
A step to make this trait object-safe.
Ref #3934 .
2022-03-21 10:45:05 +01:00
0071b85c22
refactor: make `ExecutionContextProvider` object-safe
...
Ref #3934 .
2022-03-21 10:40:53 +01:00
169fa2fb2f
refactor: make `QueryChunk` object-safe
...
This makes it way easier to dyn-type database implementations. The only
real change is that we make `QueryChunk::Error` opaque. Nobody is going
to inspect that anyways, it's just printed to the user.
This is a follow-up of #4053 .
Ref #3934 .
2022-03-18 11:40:31 +01:00
a122b1e2ca
refactor: dyn-typed DB for `query_tests` ( #4053 )
...
To test the `db::Db` as well as the `querier` with the same test
framework, they require a shared interface. Ideally this interface is
dynamically typed instead of static dispatched via generics because:
- `query_tests` already take ages to compile
- we often hold a list of scenarios and a single scenario should (in a
future PR) be able to represent both OG as well as NG
The vision here is that we basically keep the whole test setup but add
new scenarios which are NG-specific later on.
Now the issue w/ many query-related types is that they are NOT
object-safe because methods that don't take `&self` or they have
associated types that we cannot specify in general for OG and NG at the
same time.
So we need a bunch of wrappers that make dynamic dispatch possible. They
mostly call to an internal "interface" crate which is the actual `dyn`
part. The interface is currently only implemented for OG.
The scenarios currently also only contain OG databases. However,
creating a dynamic interface that can be used in all `query_tests` is
already a huge step.
Note that there are two places where we downcast the dynamic/abstract
database to `db::Db` again:
1. To create one scenario based on another and where we need to
manipulate `db::Db` with OG-specific semantics.
2. `server_benchmarks`. These contain OG databases only and there is no
point in benchmarking throw the dynamic dispatch interface because
prod (`influxdb_ioxd`) also uses static dispatch.
Ref #3934 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-18 10:11:17 +00:00
0850a93f20
refactor: make `QueryDatabase::chunks` async ( #4047 )
...
For OG we can determine the chunks w/o any IO, for NG however this might
require a few catalog queries.
This is likely not the last change of this sort, i.e. the whole schema
handling is currently sync as well.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-17 12:55:25 +00:00
5585dd3c21
refactor: switch to using DynObjectStore
...
Changes all consumers of the object store to use the dynamically
dispatched DynObjectStore type, instead of using a hardcoded concrete
implementation type.
2022-03-15 16:32:52 +00:00
1d5066c421
refactor: rename ObjectStore -> ObjectStoreImpl
...
Frees up the name for so we can use `dyn ObjectStore` throughout the
code instead of `ObjectStoreApi`.
2022-03-15 16:29:43 +00:00
2c3d30ca32
chore: Update datafusion, arrow, flight and parquet ( #4000 )
...
* chore: Update datafusion, arrow, flight and parquet
* fix: api change
* fix: fmt
* fix: update test metadata size
* fix: Update sizes in parquet test
* fix: more metadata size update
2022-03-10 12:24:47 +00:00
77f6153f72
refactor: remove `QueryDatabase::chunk_summaries` ( #3977 )
...
- This is not used by the query engine at all.
- The query engine should not care about ALL chunks but only about the
chunks it gets via `QueryDatabase::chunks` (which includes a table
name and a predicate).
- All other users of that API are NOT really query-related.
2022-03-08 11:34:26 +00:00
48908054d1
chore(deps): Bump once_cell from 1.9.0 to 1.10.0 ( #3955 )
...
* chore(deps): Bump once_cell from 1.9.0 to 1.10.0
Bumps [once_cell](https://github.com/matklad/once_cell ) from 1.9.0 to 1.10.0.
- [Release notes](https://github.com/matklad/once_cell/releases )
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md )
- [Commits](https://github.com/matklad/once_cell/compare/v1.9.0...v1.10.0 )
---
updated-dependencies:
- dependency-name: once_cell
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2022-03-07 08:54:20 +00:00
7b436d37cd
fix: correctly set nulls in string columns ( #3939 )
...
* fix: correctly set nulls in string columns
* fix: test
* test: fix
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-03-04 20:15:44 +00:00
07fdbe7c6b
fix: apply `_field` restrictions that do not match any fields ( #3903 )
...
* fix: apply field restriction correctly
* refactor: Use Vec::retain
2022-03-03 15:04:05 +00:00
3d047073b9
feat: add tracing down to the chunk level ( #3804 )
...
* refactor: wire exectution context to Deduplicator
* feat: example trace to chunk read_filter
* refactor: make execution context required
* refactor: expose metadata API
* refactor: more span context for chunk read_filter
* refactor: fix build
* refactor: push context into result stream
* refactor: make executor optional
2022-03-02 19:08:22 +00:00
33851be3a5
chore: upgrade Rust to 1.59 ( #3875 )
...
Mostly a few new clippy crates around `flat_map`, `and_then`, and
"underscore locks" (!!!):
https://rust-lang.github.io/rust-clippy/master/index.html#let_underscore_lock
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-28 15:14:19 +00:00
2a842fbb1a
feat: correctly sort data and store in catalog metadata ( #3864 )
...
* feat: respect sort order in ChunkTableProvider (#3214 )
feat: persist sort order in catalog (#3845 )
refactor: owned SortKey (#3845 )
* fix: size tests
* refactor: immutable SortKey
* test: test sort order restart (#3845 )
* chore: explicit None for sort key
* chore: test cleanup
* fix: handling of sort keys containing fields
* chore: remove unused selected_sort_key
* chore: more docs
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-25 17:56:27 +00:00
3b7d31c88a
chore(deps): Bump arrow from 9.0.2 to 9.1.0 ( #3826 )
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 9.0.2 to 9.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/9.0.2...9.1.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-23 09:25:46 +00:00
ad3868ed7c
chore(deps): Bump tokio from 1.16.1 to 1.17.0 ( #3814 )
...
* chore(deps): Bump tokio from 1.16.1 to 1.17.0
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.16.1 to 1.17.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.16.1...tokio-1.17.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* build: update workspace-hack
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom Dwyer <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 16:27:43 +00:00
69480eaa5c
feat: remove remaining table-granularity metrics ( #3808 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-22 15:02:57 +00:00
9588b43a90
fix: Make errors in rewriting return `Error` rather than a `panic` ( #3767 )
...
* test: add test for predicate errors
* fix: Return errors properly rather than panic
* fix: handle errors in influxrpc planner
* fix: appease clippy
* fix: tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-17 15:39:14 +00:00
a30803e692
chore: Update datafusion, update `arrow`/`parquet`/`arrow-flight` to 9.0 ( #3733 )
...
* chore: Update datafusion
* chore: Update arrow
* fix: missing updates
* chore: Update cargo.lock
* fix: update for smaller parquet size
* fix: update test for smaller parquet files
* test: ensure parquet_file tests write multiple row groups
* fix: update callsite
* fix: Update for tests
* fix: harkari
* fix: use IoxObjectStore::existing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-15 12:10:24 +00:00
d9f331ba2a
chore: update datafusion, stop repartitioning so aggressively ( #3633 )
...
* chore: update datafusion
* fix: Update to use new datafusion api
* chore: update expected plans
* fix: support zero output partitions
* fix: update test
* fix: Update for new DataFusion API
* fix: newly added system table
* fix: update cargo lock
2022-02-09 19:53:41 +00:00
85004831a3
refactor: extract predicate sql tests out of rust ( #3683 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-09 11:24:34 +00:00
49edfebdf6
refactor: Complete move of test scenarios into library.rs ( #3682 )
...
Co-authored-by: Edd Robinson <me@edd.io>
2022-02-09 10:46:00 +00:00
c18ad4ac97
feat: special case max timestamp range for table_names and field_columns ( #3642 )
2022-02-08 16:09:36 +00:00
8d7865496d
refactor: remove out of date comments ( #3653 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 15:14:34 +00:00
2e30483f1f
refactor: Remove predicate module from predicate crate ( #3648 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-07 14:54:07 +00:00
52b2f2e606
refactor: Move more test scenarios into library.rs ( #3639 )
2022-02-04 10:53:57 +00:00
a5045de02c
fix: remove incorrect comment ( #3622 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-03 11:06:29 +00:00
a2ed6a1b75
Merge branch 'main' into combine-non-overlapping-chunks
2022-02-02 20:47:51 +00:00
766bf9826c
refactor: remove unecessary #allow lints ( #3615 )
...
* refactor: remove unecessary #allow lints
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 20:23:08 +00:00
c4a234e83c
feat: Allow sql test runner to compare sorted output ( #3618 )
...
* refactor: Add Query struct
* feat: Implement sorted checking
* refactor: port some sql tests over
* fix: fmt
* fix: Apply suggestions from code review
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 19:59:52 +00:00
8a8de19fb5
feat: combine non-overlapping chunks without deletes
2022-02-02 16:40:30 +00:00
55daf95b48
refactor: Consolidate more query_test DbSetups ( #3607 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 15:16:56 +00:00
5d5310351b
refactor: Move query_test readme to standard location ( #3608 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-02-02 15:06:13 +00:00
62e8b41509
refactor: Start consolidating test scenarios in query_tests ( #3580 )
...
* refactor: Start consolidating test scenarios in query_tests
* fix: reference
* fix: fmt
2022-01-28 21:12:57 +00:00
21c1824a7a
refactor: remove table_names from Predicate ( #3545 )
...
* refactor: remove table_names from Predicate
* chore: fix benchmarks
* chore: review feedback
Co-authored-by: Edd Robinson <me@edd.io>
* chore: review feedback
* chore: replace Default::default with InfluxRpcPredicate::default()
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-27 14:44:49 +00:00
5488c257d1
chore: Update datafusion, upgrade to arrow/parqet/arrow-flight 8.0.0 ( #3517 )
...
* chore: Update datafusion
* chore: update to arrow 8
* fix: update to use new DataFusion APIs
* fix: update case for sortedness
* fix: cargo hakari
2022-01-27 13:33:27 +00:00
54ae5de9bf
feat: chunk pruning metrics ( #3516 )
...
Co-authored-by: Edd Robinson <me@edd.io>
2022-01-25 11:11:50 +00:00
9615feacb3
fix(InfluxQL): Support RegEx with escape sequences not supported by Rust regex ( #3502 )
...
* fix(InfluxQL): Translate unsupported meta characters
* fix: remove debugging
* fix: clippy sacrifice
* docs: Add additional background and rationale for rewriting
* fix: doc link
2022-01-21 14:40:10 +00:00
9c19cd6cc4
fix: clamp start/end of TimestampRange to min/max valid timestamp values ( #3487 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 16:08:00 +00:00
f0d50f447a
fix: Special case tag_keys with max timestamp range ( #3485 )
...
* fix: Special case tag_keys with max timestamp range
* docs: comment
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 14:14:34 +00:00
9b6e626626
chore: Update datafusion (and get fix for influxql test failure) ( #3484 )
...
* test: add tests for comparing dictionary arrays
* chore: update datafusion deps
* refactor: Update code for DataFusion API changes
* fix: update test
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-20 14:01:47 +00:00
168afb63ad
feat: add `size` methods to DML-related types
...
This will be helpful when we want to batch DML operations in memory
(e.g. when using RSKafka).
This also ensures that `MBChunk` accounts for the column names that
are stored within `MutableBatch`.
2022-01-18 13:52:31 +01:00
dd23056efd
chore: update datafusion, arrow, prost, tonic, pbjson, etc ( #3455 )
...
* chore: update datafusion, arrow, prost, tonic, etc
* fix: update pprof as well
* chore: update hakari
* fix: update pbjson
* chore: update heappy
* fix: hakari
* fix: workaround https://github.com/influxdata/influxdb_iox/issues/3458
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 17:07:15 +00:00
cdf5c21cd4
fix: Fix max timestamp value comparison in chunk metadata ( #3453 )
...
* fix: Fix max timestamp value comparison in chunk metadata
* refactor: rename contains to overlaps
Co-authored-by: Edd Robinson <me@edd.io>
2022-01-13 16:58:30 +00:00
a750da43f1
test: Reproducer for #3428 ( #3429 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-13 11:44:39 +00:00
f3f6f335a9
chore: upgrade to snafu 0.7 ( #3440 )
2022-01-11 19:22:36 +00:00
336ffd1966
refactor: Remove `Result` in QueryDatabase trait (none of the functions can fail) ( #3422 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-01-06 22:03:08 +00:00
a93ae739a9
feat: Add table_name to Partition API ( #3421 )
2022-01-06 16:38:39 +00:00
f9174c483b
refactor: Extract server::db into its own crate ( #3417 )
...
* refactor: Extract JobRegistry from the server crate
Both the server crate and a db crate that I'm about to extract depend on
JobRegistry, so to avoid making circular dependencies, extract the
JobRegistry to its own crate.
* refactor: Move db out of server into its own crate
Fixes #2821 .
2021-12-23 22:01:17 +00:00
527885f7f8
chore: Update datafusion ( #3413 )
...
* chore: Update datafusion and update code to handle timezone aware timestamps
* fix: cargo hakari
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-23 14:52:12 +00:00
43e3e76de2
chore: Merge branch 'main' into ntran/compact_os_sql_tests
2021-12-08 12:07:02 -05:00
b93b7e9304
refactor: address review comments
2021-12-08 11:50:09 -05:00
218042784f
feat: add `system.queries `system table ( #3328 )
...
* feat: Add query log and system table
* docs: Add docstrings for test normalization
2021-12-08 16:26:24 +00:00
61dd7e0ba0
chore: clean up `all_chunks_dropped.sql` ( #3337 )
2021-12-08 14:51:56 +00:00
a13c3120a6
test: sql tests and checkpoint test for compact OS chunks
2021-12-07 17:32:48 -05:00
16d8ae5e04
fix: Match tokio features to what's actually in use in each crate
...
Some crates listed features they don't use; other crates ware relying on
feature flags enabled by something else. I tested these changes by
disabling the workspace hack crate and testing each crate.
2021-12-06 09:37:16 -05:00
02c297e850
fix: Always specify the parking_lot feature of tokio to get potential perf boost
2021-12-06 09:37:15 -05:00
1a899b939e
fix: Remove redundant closures identified by clippy
...
https://rust-lang.github.io/rust-clippy/master/index.html#redundant_closure
2021-12-02 11:52:02 -05:00
9e8639f230
chore: Update DataFusion pin ( #3279 )
...
* chore: Update DataFusion pin
* fix: Update for new DF API
* fix: update plan output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-12-02 12:42:28 +00:00
ac20fecf6f
refactor: remove write buffer producer logic from `Db`
...
As a side effect, writing to database no longer requires `async`.
Closes #2243 .
2021-11-26 12:02:27 +01:00
48dd1ae757
fix: ensure time column not returned
2021-11-24 16:23:43 +00:00
c0a2732c77
test: update read_window_aggregate tests
2021-11-23 14:15:06 +00:00
71a0560300
test: update read_group tests
2021-11-23 14:15:06 +00:00
e54e7825d3
test: update read_filter tests
2021-11-23 14:15:06 +00:00
50e9e02ff7
chore: Update datafusion ( #3188 )
...
* chore: Update datafusion
* fix: Update for LogicalPlan changes
* fix: Update explain plan output
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-23 12:23:23 +00:00
9fd4a560f5
feat: Results of running cargo hakari manage-deps
2021-11-19 09:21:57 -05:00
a2454b542d
fix: Small cleanups in Cargo.tomls ( #3160 )
...
* fix: Add tokio rt-multi-thread feature so cargo test -p client_util compiles
* fix: Alphabetize dependencies
* fix: Add the data_types_conversions feature to get tests passing
* fix: Remove dev dependencies already listed under normal dependencies
* fix: Make sure the workspace is using the new resolver
2021-11-18 22:26:33 +00:00
8155747735
feat: add write buffer delete encoding ( #2731 ) ( #3127 )
...
* feat: add write buffer delete encoding (#2731 )
* chore: fix doc
* chore: review feedback
* chore: review feedback
* chore: fmt
* chore: review feedback
2021-11-17 16:12:19 +00:00
3d091208af
refactor: move delete predicate into data_types ( #2731 ) ( #3094 )
...
* refactor: move delete predicate into dml (#2731 )
* refactor: move DeletePredicate to data_types
* chore: fix doc
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-15 10:28:58 +00:00
71731524c4
test: add group by tests
2021-11-08 10:46:40 -05:00
abbfafcabd
chore: merge main to branch
2021-11-08 09:28:29 -05:00
97206b13cb
fix: statistics for max/min(time) should have data type timstamp
2021-11-05 18:11:54 -04:00
60f0deaf1e
feat: remove flatbuffer entry ( #3045 )
2021-11-05 20:19:24 +00:00
a252b81baa
test: add negative test for CREATE EXTERNAL TABLE ( #3051 )
...
* test: add negative test for CREATE EXTERNAL TABLE
* fix: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-11-05 17:43:08 +00:00
89699cf0de
chore: use latest DataFusion to fix the min/max(dictionary string) bug
2021-11-04 11:04:03 -04:00
5e86989990
fix: Only use timestamps in first/last when there is a corresponding … ( #2988 )
...
* fix: Only use timestamps in first/last when there is a corresponding value
* docs: Fix broken English in comment
* docs: clarify expectations in test
2021-11-01 18:54:43 +00:00
c540b40f05
chore(deps): bump tokio from 1.12.0 to 1.13.0
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.12.0...tokio-1.13.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 11:21:59 +00:00
d2c2143277
fix: wrong results in read_filter with "complex" OR predicate and IS_NULL ( #2977 )
...
* fix: Allow evaluating general purpose predicates in gRPC by rewriting missing columns to null
* fix: update commands and add some additional tests
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-29 14:30:28 +00:00
6760212e86
Merge branch 'main' into ntran/group
2021-10-26 21:25:57 +00:00
7b3b92b161
chore: make comments clearer
2021-10-26 16:52:45 -04:00
682499d4a7
chore: make comments clearer
2021-10-26 16:50:27 -04:00
8c43b24745
chore: cleanup
2021-10-26 16:46:50 -04:00
46cb4df14d
fix: now no rows will be return for read_group if the table has no data or all its data is soft deleted or the table does not exist
2021-10-26 16:40:36 -04:00
5451a49620
fix: flaky `test_query_cancellation_slow_store` ( #2966 )
2021-10-26 15:58:28 +00:00
7cd56cbc56
Merge remote-tracking branch 'origin/main' into er/fix/flux/2691
2021-10-25 13:41:08 -04:00
bc7244c48e
chore: use Rust edition 2021
2021-10-25 10:58:20 +02:00
52cf1a85b9
fix(metadata): Do not report table_names for tables that have no non-null values that match predicate ( #2947 )
...
* fix(metadata): Do not report table_names for tables that have no non-null values that match predicate
* fix: make apply_predicate_to_metadata precise
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-24 11:03:30 +00:00
3c87cc3747
fix: make all test scenarios have data deleted after chunks are moved
2021-10-23 07:56:15 -04:00
28962038c1
fix: properly handle null values in table_name plan ( #2946 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-23 10:35:02 +00:00
a6fb7bf2b5
test: fix read_group test case to make it work with all scenarios
2021-10-22 17:22:30 -04:00
eb71102d82
Merge branch 'main' into er/fix/flux/2691
2021-10-22 20:21:22 +01:00
e733562da8
chore: fix comment
2021-10-22 12:39:56 -04:00
f9dedd78da
test: Make test outputs consistent with the ones we send back to client
2021-10-22 12:19:30 -04:00
44e0ef757f
Merge branch 'main' into er/fix/flux/2691
2021-10-22 13:56:27 +01:00
b3242277eb
test: test_field_name_plan_with_delete is fixed
2021-10-22 13:07:36 +01:00
fc7ce3535d
refactor: fix _value filtering for non-selector aggregates
2021-10-21 22:28:41 +01:00
f8489fc774
refactor: ensure _value filtering works read_filter
2021-10-21 22:28:41 +01:00
3e687377e0
test: add test for defect #2691
2021-10-21 22:24:03 +01:00
f7ca80e29f
test: ensure query cancellation (somewhat) works ( #2931 )
...
* feat: enable reconfiguration of in-use throttled store
This is handy for tests for which a part should run "normal" and another
one should be throttled/blocked.
* feat: keep track of the number of tasks within a `DedicatedExecutor`
* test: ensure query cancellation (somewhat) works
We cannot really test that query cancellation finishes all subtasks
because _tokio_ doesn't provide sufficient stats / inspection, at least
as long we don't want to rely heavily on _tokio_ tracing. So let's at
least check that tasks from the dedicated executors are pruned properly.
For all other regressions we need to add unit tests to the affected
components. See for example:
- https://github.com/apache/arrow-datafusion/issues/1103
- https://github.com/apache/arrow-datafusion/pull/1105
- https://github.com/apache/arrow-datafusion/pull/1112
- https://github.com/apache/arrow-datafusion/pull/1121
Closes #2027 .
2021-10-21 19:10:58 +00:00
aecbdc0468
chore: merge main to branch
2021-10-20 17:09:02 -04:00
97d5760347
feat: support delelete for table_names
2021-10-20 15:58:25 -04:00
ee2ca8fc32
fix(read_group): Support grouping on `_start` and `_stop` ( #2918 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 15:16:47 +00:00
5059d9cf8a
test: add _field filter coverage to window_aggregate
2021-10-20 14:59:27 +01:00
3db48d1460
test: add _field filter coverage to read_group
2021-10-20 14:59:27 +01:00
f397f0b4ac
test: add test for defect
2021-10-20 14:59:27 +01:00
9974a5364c
chore(security): Replace prettytable with comfy-table ( #2905 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 10:44:36 +00:00
ce0127a1f7
feat: MutableBatch write API ( #2090 ) ( #2724 ) ( #2882 )
...
* feat: MutableBatch write API (#2090 ) (#2724 )
* chore: fix lint
* fix: handle dictionaries with unused mappings
* chore: review feedback
* chore: further review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-20 08:44:14 +00:00
b55ca06fe3
fix: read_window_aggregate overflow ( #2908 )
2021-10-19 19:20:39 +00:00
e664799593
feat: support delete for tag_values
2021-10-19 14:18:01 -04:00
254a842eed
Merge branch 'main' into ntran/table_names
2021-10-19 17:53:25 +00:00
d5cffb5f54
fix(read_window_aggregates): return aggregates as integer rather than unsigned ( #2906 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:47:15 +00:00
cabb007956
chore: Merge branch 'main' into ntran/table_names
2021-10-19 13:22:28 -04:00
a82dc6f5f0
chore: Update datafusion + arrow ( #2903 )
...
* chore: Update datafusion to latest, arrow to 6.0.0
* fix: Update tests
* fix: bubble internal error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-19 17:14:08 +00:00
716515fc5a
fix: turn ignore back on for table_name tests as it is not ready yet
2021-10-19 10:38:25 -04:00
ea85d6478e
chore: remove code changes for table_names abecause normal plan needs to be implemented first
2021-10-19 10:36:10 -04:00
a364bdfb5f
chore: remove unused `query_tests` => `chrono` dep
2021-10-19 14:45:56 +02:00
afa6e50c9c
feat: make tag_keys work with delete
2021-10-18 15:36:19 -04:00
30c3d1e001
test: Add read_group test for missing fields ( #2881 )
...
* test: Add read_group test
* fix: Update to use o2 table
2021-10-18 18:43:52 +00:00
f5a84122e3
feat: Support grouping by _field and _measurement ( #2874 )
...
* feat: Support grouping by _field and _measurement
* fix: clippy
* fix: doclink
* refactor: rename SeriesOrGroup --> Either
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-18 15:32:24 +00:00
9244e9fc4e
test: Delete tests for Influxrpc queries
2021-10-15 17:26:36 -04:00
beaf77cecf
refactor: move Series translation logic into query crate, update gRPC tests ( #2852 )
...
* refactor: move Series translation logic into query crate
* refactor: update grpc_tests to use new display
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-15 11:06:40 +00:00
b2d265dc51
chore: run format after accepting reviewer's suggestions
2021-10-14 17:31:42 -04:00
39a556c5eb
chore: Apply suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-14 17:21:11 -04:00
6a5aa00e2c
chore: make the comments and names clearer
2021-10-14 17:11:37 -04:00
993c6173d1
Merge branch 'main' into ntran/grpc_storage
2021-10-14 15:28:05 +00:00
69d1253240
chore: Merge branch 'ntran/grpc_storage' of https://github.com/influxdata/influxdb_iox into ntran/grpc_storage
2021-10-14 11:23:29 -04:00
faf65f38cc
refactor: address review comments
2021-10-14 11:23:20 -04:00
08f1831aef
refactor: Apply suggestions from code review
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-14 09:25:44 -04:00
693c878781
refactor: update test
2021-10-14 09:23:29 +01:00
75caa4aad8
test: add test duplicating defect input
2021-10-14 09:23:29 +01:00
8dd9dcce01
test: verify if all scenarios are created correctly and add a few delete tests for read_filter
2021-10-13 17:21:03 -04:00
f85d6d5da8
chore: cleanup
2021-10-12 17:23:01 -04:00
401124548c
chore: ignore failed newly added test due to bug
2021-10-12 17:20:17 -04:00
144ce77e39
chore: merge main to branch
2021-10-12 15:59:57 -04:00
459dd46ae9
refactor: move delete tests to .sql
2021-10-12 15:49:23 -04:00
035654b4f9
refactor: do not rebuild query_test when .sql or .expected files change ( #2816 )
...
* feat: Do not rebuild query_tests if .sql or .expected change
* feat: Add CI check
* refactor: move some sql tests to .sql files
* tests: port tests / expected results to data files
* fix: restore old name check-flatbuffers
2021-10-12 19:34:54 +00:00
9b6726b99c
refactor: rename to a more general name function
2021-10-12 10:34:55 -04:00
0b4ae95ca4
refactor: exhaust scenarios for one-chunk test
2021-10-11 17:47:41 -04:00
b39e01f7ba
feat: migrate PersistenceWindows to TimeProvider ( #2722 ) ( #2798 )
2021-10-11 20:40:00 +00:00
fbf5539336
chore: merge main to branch
2021-10-11 10:47:10 -04:00
afe34751e7
refactor: split out schema crate ( #2781 )
...
* refactor: split out schema crate
* chore: fix doc
2021-10-11 09:45:08 +00:00
f7475322a6
chore: merge main to branch, resolve conflicts, and discover an inconsitent bug
2021-10-08 15:50:46 -04:00
f2cdb9531f
chore: cleanup
2021-10-08 14:52:15 -04:00
adbcd85c26
fix: fully fix 2745
2021-10-08 14:37:34 -04:00
2072b4066e
feat: Implement support for `_measurement` predicate in gRPC plans ( #2772 )
...
* feat: Implement filtering for _measurement in general purpose gRPC plans
* docs: fixup docstrings
* fix: fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-08 17:25:33 +00:00
22d6f11bea
fix: add cols of delete predicates into the schema of scanning columns
2021-10-07 17:37:34 -04:00
63d74be490
refactor: make `ChunkId` a UUID
2021-10-07 10:23:27 +02:00
de148337e8
fix: half way fix the bug to inlcude schema of column in delete predicate into the schema of IOx scan to avoid missing reading columns
2021-10-06 17:43:48 -04:00
efa2316626
fix: do not sort the output of read_group with no group keys ( #2755 )
2021-10-06 18:59:58 +00:00
bd93b411c7
chore: cleanup
2021-10-06 10:57:51 -04:00
1aee2a49eb
Merge branch 'main' into ntran/no_use_stats
2021-10-06 14:06:01 +00:00
65a02f7085
refactor: Apply suggestions from code review
...
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-10-06 10:04:28 -04:00
ce5b24e65d
refactor: use DateTime<Utc> in PersistenceWindows ( #2722 ) ( #2743 )
...
* refactor: use DateTime<Utc> in PersistenceWindows (#2722 )
* chore: fix benchmark
* chore: fmt
* chore: review feedback
2021-10-06 09:39:32 +00:00
055e69439d
test: fix auto created tests
2021-10-05 18:11:27 -04:00
aa64daca86
feat: dDisable using statistics to query data if there are soft deleted rows
2021-10-05 17:52:32 -04:00
d0929e3a34
feat: persist no chunks ( #2712 ) ( #2718 )
...
* feat: persist no chunks (#2712 )
* fix: persist partition
* fix: chunk ordering test
* chore: fix logical conflict
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-05 15:18:35 +00:00
10c1a72402
refactor: remove unused fields from `DeletePredicate`
2021-10-05 09:29:24 +02:00
1856d7184c
fix: make stop time inclusive
2021-10-04 14:14:16 -04:00
64cd0cb3ce
chore: cleanup
2021-10-04 12:43:35 -04:00
ba96b20b1f
test: turn on tests of delete all
2021-10-04 12:28:49 -04:00
75ac6e8646
refactor: make `DeletePredicate::range` non-optional
2021-10-04 16:36:20 +02:00
5a5a929b9e
refactor: introduce `DeletePredicate`
...
`DeletePredicate` is a simpler version of `Predicate` that is based on
IOx `DeleteExpr` instead of the full-blown DataFusion `Expr`. This will
allow us to do a couple of things (in follow-up changes):
- Order and de-duplicate delete predicates
- Normalize predicates
- Infallible serialization
- Smaller memory footprint
Note that this change only affects delete expressions. Query expressions
that are supported via the API are not changed. The query subsystem also
still uses the full-featured expressions/predicates (delete
expressions/predicates are converted to the more powerful DataFusion
version on-the-fly).
2021-10-04 16:36:20 +02:00
7ab10daa19
Merge branch 'main' into dependabot/cargo/arrow-5.5.0
2021-10-04 12:58:29 +01:00
e8eab2cc97
feat: allow compaction and persistence to retun no chunk ( #2664 ) ( #2700 )
...
* feat: allow compaction and persistence to retun no chunk (#2664 )
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-10-04 10:54:47 +00:00
d1f5209869
chore(deps): bump arrow from 5.4.0 to 5.5.0
...
Bumps [arrow](https://github.com/apache/arrow-rs ) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/5.5.0/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/5.4.0...5.5.0 )
---
updated-dependencies:
- dependency-name: arrow
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-04 08:55:38 +00:00
b402423e9e
feat: remove move lifecycle action ( #2674 )
...
* feat: remove move_chunk lifecycle action
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-30 16:58:05 +00:00
Carol (Nichols || Goulding)
Andrew Lamb
Carol (Nichols || Goulding)
Nga Tran
Marco Neumann
dependabot[bot]
kodiakhq[bot]
Dom Dwyer
Edd Robinson
Raphael Taylor-Davies
Andrew Lamb
Raphael Taylor-Davies
Nga Tran