115f007317
refactor: Use DataFusion `Expr` instead of our own custom wrapper for `ValueExpr` ( #4440 )
...
* refactor: Use DataFusion `Expr` instead of custom wrapper for BinaryExprs
* fix: apply code review suggestions
* fix: more code review suggestions
2022-04-27 19:20:15 +00:00
59f6556483
fix: do not create empty batches in ingester ( #4443 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 17:52:22 +00:00
a195973cfb
chore(deps): Bump clap from 3.1.11 to 3.1.12 ( #4406 )
...
* chore(deps): Bump clap from 3.1.11 to 3.1.12
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.11 to 3.1.12.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.11...v3.1.12 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: Update tests now that the clap crossed-streams bug has been fixed
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 17:43:55 +00:00
468f576698
fix: do not prune all-NULL columns from sort keys ( #4442 )
2022-04-27 16:06:26 +00:00
bea4556749
chore: Ignore flaky OG end to end test ( #4441 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 15:17:46 +00:00
6d2a8256ba
feat: Add window_bounds to IOx Function Registry (2.5/3) ( #4432 )
...
* feat: Add window_bounds to IOx Function Registry
* refactor: Prepare for lib tests
* test: Add explicit tests for plumbing
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 15:08:38 +00:00
663944a2a9
chore(deps): Bump all rusoto crates from 0.47.0 to 0.48.0 ( #4439 )
...
* chore(deps): Bump rusoto_credential from 0.47.0 to 0.48.0
Bumps [rusoto_credential](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_credential
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_core from 0.47.0 to 0.48.0
Bumps [rusoto_core](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): Bump rusoto_s3 from 0.47.0 to 0.48.0
Bumps [rusoto_s3](https://github.com/rusoto/rusoto ) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/rusoto/rusoto/releases )
- [Changelog](https://github.com/rusoto/rusoto/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rusoto/rusoto/compare/rusoto-v0.47.0...mock-v0.48.0 )
---
updated-dependencies:
- dependency-name: rusoto_s3
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-27 14:58:40 +00:00
bd0bae13ce
fix: extend + harden querier `ensure_schema` ( #4429 )
...
- only convert dictionary types that we really want to convert (instead
of blindly converting all types)
- handle missing / NULL columns
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 12:49:59 +00:00
5e91a3b909
chore: Update datafusion ( #4436 )
2022-04-27 11:25:10 +00:00
f0ceca985b
feat: Implement IOx Function Registry for regex_match/regex_not_match ( #4431 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-27 10:33:19 +00:00
b01a5bb0a9
chore(deps): Bump httparse from 1.7.0 to 1.7.1 ( #4434 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.7.0...v1.7.1 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-27 08:12:23 +00:00
fa2c1febf4
feat: use stored partition sort key to deduplicate data ( #4360 )
...
* feat: use stored sort key to deduplicate data
* refactor: verify if one is a super sort key of the other
* test: unit tests for scan and deduplication plans
* fix: typo
* refactor: refactor and add comments
* feat: cache partition sort key to read during planning as needed
* test: tests for query plans with different overlap groups
* chore: cleanup
* chore: resolve merge conflicts
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 20:36:32 +00:00
9e91af4501
refactor: Move IOx UDfs into a Function Registry (1/3) ( #4428 )
...
* refactor: Move all UDF implementations to query_function crate
* refactor: Move regex udf to query_functions
* refactor: Move functions out of query
* fix: lints, imports
* chore: Run cargo hakari tasks
* fix: clipy + benches
* fix: reduce borrowing and fix clippy
* fix: moar clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 17:30:27 +00:00
11f87cffdd
fix: memorize max persisted tombstone ( #4430 )
2022-04-26 16:13:09 +00:00
a72544f42d
chore(deps): Bump tower-http from 0.2.5 to 0.3.0 ( #4425 )
...
* chore(deps): Bump tower-http from 0.2.5 to 0.3.0
Bumps [tower-http](https://github.com/tower-rs/tower-http ) from 0.2.5 to 0.3.0.
- [Release notes](https://github.com/tower-rs/tower-http/releases )
- [Commits](https://github.com/tower-rs/tower-http/compare/tower-http-0.2.5...tower-http-0.3.0 )
---
updated-dependencies:
- dependency-name: tower-http
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 15:05:42 +00:00
bd600bbac6
refactor: allow ingester to be integrated into query tests ( #4427 )
...
* refactor: improve `IngesterData` public interface
* feat: impl `Debug` for `Test{Namespace,Sequencer}`
* refactor: trait interface for `LifecyleHandle`
This is required to mock the lifecycle for query tests.
* refactor: trait for partitioner
2022-04-26 13:44:30 +00:00
401009a7b6
Merge pull request #4419 from influxdata/dom/http-service-protection-limit
...
feat(router2): http service protection limit
2022-04-26 10:21:57 +00:00
8a208a814a
feat: emit http_request_limit_rejected metric
...
Add the "http_request_limit_rejected" metric that is incremented once
each time a request is dropped due to the simultaneous request service
protection limit being exceeded.
This metric will allow us to effectively alert on router saturation, and
the increase rate/second will help inform us in how much capacity needs
adding.
2022-04-26 11:13:26 +01:00
fb777e7e51
feat(router2): configurable max HTTP requests
...
Adds a CLI / env configuration option controlling the maximum number of
simultaneous HTTP requests in the router.
2022-04-26 11:13:25 +01:00
94358af65c
feat(router2): simultaneous HTTP request limit
...
This commit adds a service protection limit, allowing the router to
restrict the number of simultaneous HTTP requests to a hard-coded limit
of 200.
By dropping additional requests once the router is serving the
configured maximum, the router is prevented from accepting more and more
requests that stack up waiting on the catalog / postgres connection
pool. Currently buffering all these stalled connections consumes so much
memory the router is OOM killed. After this commit, a portion of
requests will be satisfied and the rest will be dropped early with a
HTTP 503 Service Unavailable response.
2022-04-26 11:13:24 +01:00
2337935660
test: chunks in ingester stage ( #4415 )
...
* refactor: document and improve `MockIngesterConnection`
* refactor: split `OldOneMeasurementFourChunksWithDuplicates` for `EXPLAIN` queries
* fix: mark "IngsterPartition" chunks as unsorted
* fix: "group by" queries may require sorted comparison
* refactor: re-export a few more types from querier
* fix: ensure that test parquet files are de-duped
* test: chunks in ingester stage
* docs: explain test code
2022-04-26 07:55:19 +00:00
4b47d723b1
refactor: Rename time to iox_time ( #4416 )
...
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-26 00:19:59 +00:00
0a440bb638
refactor: grouping overlaps now uses the same overlap function in both compactor and deduplication ( #4420 )
...
* refactor: grouping overlaps is now use the same overlap function in both compactor and deduplication
* chore: commit missing file
* chore: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-25 20:32:51 +00:00
e3caf24954
feat: Rewrite the NG end-to-end metrics test ( #4404 )
...
* refactor: Expose data generation tool for wider use
* feat: Add a step for retrieving the server metrics
* refactor: Copy the OG end-to-end metrics test to NG
* feat: Rewrite the NG end-to-end metrics test
This is still broken because the the row timestamp metrics don't exist
in NG.
* fix: Test metrics relevant to NG
* refactor: Move the data generator to the test helper crate
* refactor: Extract a ReadFilter request builder into the test helper crate
* refactor: Make test helper request builder able to build other gRPC requests
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
2022-04-25 19:47:56 +00:00
86e8f05ed1
fix: make all catalog IDs 64bit ( #4418 )
...
Closes #4365 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-25 16:49:34 +00:00
4c94e46642
chore(deps): Bump croaring from 0.5.2 to 0.6.0 ( #4408 )
...
* chore(deps): Bump croaring from 0.5.2 to 0.6.0
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.5.2 to 0.6.0.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.5.2...0.6.0 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: croaring 0.6.0 compat
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
2022-04-25 16:41:08 +00:00
8be545a71c
Merge pull request #4405 from influxdata/cn/e2e-grpc-write
...
feat: Write end-to-end tests for the gRPC write API in NG
2022-04-25 15:24:57 +00:00
ec0446e480
Merge branch 'main' into cn/e2e-grpc-write
2022-04-25 15:16:20 +00:00
f482286ac9
fix: Update function name that was fixed upstream
2022-04-25 11:04:37 -04:00
f5f80e879e
test: add benchmarks for addressable heap ( #4201 )
2022-04-25 14:37:29 +00:00
35690d390b
fix: Simplify a comment
2022-04-25 10:37:23 -04:00
e2ebeff39c
feat: Add write tokens to the gRPC write endpoint response
...
As a header in the same way the HTTP write endpoint does.
Fixes #4208 .
2022-04-25 09:46:13 -04:00
0bda66a01d
feat: Write end-to-end tests for the gRPC write API in NG
...
Fixes #3941 .
2022-04-25 09:46:13 -04:00
d963110842
feat: group chunk overlaps based on time range only ( #4389 )
...
* feat: overlap for NG querier
* chore: cleanup
* refactor: address review comments
* fix: typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-25 13:32:07 +00:00
77d8967c8e
Merge pull request #4409 from influxdata/dom/bench-lp-parse
...
feat: benchmark LP -> MutableBuffer conversion
2022-04-25 11:24:12 +00:00
4174c88bc0
feat: benchmark LP -> MutableBuffer conversion
...
Adds a benchmark that measures the time taken to parse the
test_fixtures/metrics.lp dataset, and convert it into a MutableBuffer
instance.
2022-04-25 11:44:35 +01:00
d75f2d2791
Merge pull request #4398 from influxdata/dom/fix-mb-panic
...
fix: handle LP lines with duplicated fields/tags
2022-04-25 10:39:55 +00:00
2002875157
Merge branch 'main' into dom/fix-mb-panic
2022-04-25 10:31:48 +00:00
86b43e1eb3
docs: link to relevant tickets in LP test cases
2022-04-25 11:29:57 +01:00
71da70e6fe
test: cover all string field types
...
Adds tests that cover EscapedStr::SingleSlice in addition to the
existing EscapedStr::CopiedValue variants. Remove 1 space from assert.
2022-04-25 11:28:52 +01:00
376b95a387
chore(deps): Bump nix from 0.24.0 to 0.24.1 ( #4407 )
...
Bumps [nix](https://github.com/nix-rust/nix ) from 0.24.0 to 0.24.1.
- [Release notes](https://github.com/nix-rust/nix/releases )
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.0...v0.24.1 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-25 08:53:30 +00:00
f92fa69c8c
fix: Typo in test helper method name ( #4402 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 20:21:30 +00:00
117569184e
feat: Port end-to-end logging test to NG ( #4400 )
...
* feat: Copy end-to-end logging test to NG
This was created with:
cp influxdb_iox/tests/end_to_end_cases/influxdb_ioxd.rs influxdb_iox/tests/end_to_end_ng_cases/logging.rs
* feat: Port logging test to NG end-to-end tests
And re-enable it, it was ignored.
* fix: Specify that an in-memory catalog should be used for the logging test
* fix: Check for gRPC instead of HTTP
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 18:05:43 +00:00
680742103a
refactor: use std::mem::discriminant
2022-04-22 17:37:25 +01:00
14cb2f5674
fix: less async shenanigans in end to end tests ( #4384 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 14:38:57 +00:00
ff902c40d2
chore: port debug end to end tests to NG ( #4393 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 14:30:09 +00:00
cf200e509e
fix: handle LP lines with duplicated fields/tags
...
This commit allows the LP consumer to correctly handle line protocol
writes that duplicate one or more fields or tags within a single line:
table v=2,bananas=42,v=3,platanos=24
▲ ▲
└───────┬──────┘
│
duplicate field "v"
This change implements the following logic when processing fields:
IF field is duplicated
IF all duplicate field values are of the same data type
use last occurrence ("last write wins")
ELSE
return an error
Any duplication of tags is rejected, and use of a field name as both a
field and a tag is remains forbidden (unchanged in this PR, a previously
agreed breaking change from TSM).
See https://github.com/influxdata/influxdb_iox/issues/4326 for context.
2022-04-22 14:40:51 +01:00
c6441b2df1
chore: Update datafusion again ( #4396 )
...
* chore: Update datafusion again
* fix: restore clap
2022-04-22 13:40:20 +00:00
e8bfd7a537
chore(deps): Bump clap from 3.1.10 to 3.1.11 ( #4390 )
...
* chore(deps): Bump clap from 3.1.10 to 3.1.11
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.10 to 3.1.11.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.10...v3.1.11 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: update tests for changes to clap
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 11:15:48 +00:00
36ef122c89
Merge pull request #4391 from influxdata/crepererum/ingester_partitions_summary
...
feat: Provide basic summaries for `IngesterPartition`
2022-04-22 11:02:23 +00:00
Andrew Lamb
Marco Neumann
dependabot[bot]
Carol (Nichols || Goulding)
Nga Tran
kodiakhq[bot]
Dom Dwyer
二手掉包工程师
Jake Goulding
Carol (Nichols || Goulding)