35690d390b
fix: Simplify a comment
2022-04-25 10:37:23 -04:00
e2ebeff39c
feat: Add write tokens to the gRPC write endpoint response
...
As a header in the same way the HTTP write endpoint does.
Fixes #4208 .
2022-04-25 09:46:13 -04:00
0bda66a01d
feat: Write end-to-end tests for the gRPC write API in NG
...
Fixes #3941 .
2022-04-25 09:46:13 -04:00
d963110842
feat: group chunk overlaps based on time range only ( #4389 )
...
* feat: overlap for NG querier
* chore: cleanup
* refactor: address review comments
* fix: typo
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-25 13:32:07 +00:00
77d8967c8e
Merge pull request #4409 from influxdata/dom/bench-lp-parse
...
feat: benchmark LP -> MutableBuffer conversion
2022-04-25 11:24:12 +00:00
4174c88bc0
feat: benchmark LP -> MutableBuffer conversion
...
Adds a benchmark that measures the time taken to parse the
test_fixtures/metrics.lp dataset, and convert it into a MutableBuffer
instance.
2022-04-25 11:44:35 +01:00
d75f2d2791
Merge pull request #4398 from influxdata/dom/fix-mb-panic
...
fix: handle LP lines with duplicated fields/tags
2022-04-25 10:39:55 +00:00
2002875157
Merge branch 'main' into dom/fix-mb-panic
2022-04-25 10:31:48 +00:00
86b43e1eb3
docs: link to relevant tickets in LP test cases
2022-04-25 11:29:57 +01:00
71da70e6fe
test: cover all string field types
...
Adds tests that cover EscapedStr::SingleSlice in addition to the
existing EscapedStr::CopiedValue variants. Remove 1 space from assert.
2022-04-25 11:28:52 +01:00
376b95a387
chore(deps): Bump nix from 0.24.0 to 0.24.1 ( #4407 )
...
Bumps [nix](https://github.com/nix-rust/nix ) from 0.24.0 to 0.24.1.
- [Release notes](https://github.com/nix-rust/nix/releases )
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.0...v0.24.1 )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-25 08:53:30 +00:00
f92fa69c8c
fix: Typo in test helper method name ( #4402 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 20:21:30 +00:00
117569184e
feat: Port end-to-end logging test to NG ( #4400 )
...
* feat: Copy end-to-end logging test to NG
This was created with:
cp influxdb_iox/tests/end_to_end_cases/influxdb_ioxd.rs influxdb_iox/tests/end_to_end_ng_cases/logging.rs
* feat: Port logging test to NG end-to-end tests
And re-enable it, it was ignored.
* fix: Specify that an in-memory catalog should be used for the logging test
* fix: Check for gRPC instead of HTTP
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 18:05:43 +00:00
680742103a
refactor: use std::mem::discriminant
2022-04-22 17:37:25 +01:00
14cb2f5674
fix: less async shenanigans in end to end tests ( #4384 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 14:38:57 +00:00
ff902c40d2
chore: port debug end to end tests to NG ( #4393 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 14:30:09 +00:00
cf200e509e
fix: handle LP lines with duplicated fields/tags
...
This commit allows the LP consumer to correctly handle line protocol
writes that duplicate one or more fields or tags within a single line:
table v=2,bananas=42,v=3,platanos=24
▲ ▲
└───────┬──────┘
│
duplicate field "v"
This change implements the following logic when processing fields:
IF field is duplicated
IF all duplicate field values are of the same data type
use last occurrence ("last write wins")
ELSE
return an error
Any duplication of tags is rejected, and use of a field name as both a
field and a tag is remains forbidden (unchanged in this PR, a previously
agreed breaking change from TSM).
See https://github.com/influxdata/influxdb_iox/issues/4326 for context.
2022-04-22 14:40:51 +01:00
c6441b2df1
chore: Update datafusion again ( #4396 )
...
* chore: Update datafusion again
* fix: restore clap
2022-04-22 13:40:20 +00:00
e8bfd7a537
chore(deps): Bump clap from 3.1.10 to 3.1.11 ( #4390 )
...
* chore(deps): Bump clap from 3.1.10 to 3.1.11
Bumps [clap](https://github.com/clap-rs/clap ) from 3.1.10 to 3.1.11.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.1.10...v3.1.11 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: update tests for changes to clap
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 11:15:48 +00:00
36ef122c89
Merge pull request #4391 from influxdata/crepererum/ingester_partitions_summary
...
feat: Provide basic summaries for `IngesterPartition`
2022-04-22 11:02:23 +00:00
a757b8e935
Merge branch 'main' into crepererum/ingester_partitions_summary
2022-04-22 10:53:54 +00:00
c9c41f4aed
refactor: use `info!` rather `println!` in end to end tests ( #4380 )
...
* refactor: use `info!` rather `println!` in end to end tests
* chore: change from println to info in end_to_end_ng_cases too
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 10:01:20 +00:00
99f6fb5f59
feat: calculate summaries for `IngesterPartition`
2022-04-22 10:21:14 +02:00
c0ed688043
refactor: Split influxrpc end to end tests into smaller modules ( #4382 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 07:53:45 +00:00
7907a2bae3
fix: column summary conversion for "unknown" TS ( #4379 )
...
* fix: column summary conversion for "unknown" TS
Both IOx and DataFusion have the same data model for min/max statistics:
`Option<Option<i64>>` (or any other inner type)
The interpretation is:
1. **`None`:** Value unknown.
2. **`Some(None)`:** Value known to be NULL.
3. **`Some(Some(x))`:** Value known and non NULL.
The bug was that during the conversion from the IOx statistics type to
the DataFusion statistics type for timestamps, case 1 was converted into
case 2.
Up until now this didn't make a difference between timestamps were
basically known all the time, but during the development of NG there are
cases where the timestamps are unknown (this might change, but the query
engine should be correct w/o assuming that).
* docs: explain test
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-22 07:44:55 +00:00
6f4c5506d8
chore(deps): Bump tracing-log from 0.1.2 to 0.1.3 ( #4388 )
...
Bumps [tracing-log](https://github.com/tokio-rs/tracing ) from 0.1.2 to 0.1.3.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-log-0.1.2...tracing-log-0.1.3 )
---
updated-dependencies:
- dependency-name: tracing-log
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 21:25:48 +00:00
a6e3da4dc3
Merge pull request #4386 from influxdata/cn/port-tracing-test
...
feat: Port end-to-end tracing tests to NG end-to-end tests
2022-04-21 21:16:14 +00:00
8250c77253
Merge branch 'main' into cn/port-tracing-test
2022-04-21 21:07:18 +00:00
e67cc9dbce
chore: Update datafusion again ( #4385 )
...
* chore: Update datafusion
* fix: Update imports
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 21:05:16 +00:00
2087dad1f7
feat: Port tracing tests to NG end-to-end tests
2022-04-21 16:22:43 -04:00
02692751c7
feat: Copy end-to-end tracing tests to NG end-to-end tests
...
This doesn't compile at all, but this is a straight cp so should make it
a bit easier to see what has changed in the port in the next commit.
2022-04-21 16:20:34 -04:00
d88cf324fa
feat: Add a method to create an all-in-one MiniCluster from one TestConfig
2022-04-21 16:18:55 -04:00
bf8603cc31
feat: Add test config methods to configure tracing
2022-04-21 16:18:43 -04:00
3499207911
feat: Make the end-to-end UDP listener available in NG end-to-end tests
2022-04-21 15:16:57 -04:00
c7a1c496cf
fix: incorrect overlapped grouping ( #4082 )
...
* test: Failing test for finding overlapped groups
* test: Failing test for query overlap too :(
* fix: Group parquet files overlapped by time correctly
Inspired by https://towardsdatascience.com/overlapping-time-period-problem-b7f1719347db
Not sure what the real name for this algorithm is
* refactor: Group items without an intermediate hashmap needed
* chore: cleanup
Co-authored-by: NGA-TRAN <nga-tran@live.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 18:51:30 +00:00
d41086ac7f
refactor: Move Connection in ServerFixture to make TestServer shareable ( #4381 )
...
* refactor: Move Connection in ServerFixture to make TestServer shareable
* fix: Update docstrings
* fix: restore panic on previous error
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 18:15:53 +00:00
abd005e0c2
refactor: Consolidate cluster creation in end to end tests ( #4373 )
2022-04-21 18:03:17 +00:00
cc9205024c
test: Begin porting influxrpc tests to ng end to end tests ( #4372 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 16:24:45 +00:00
f302ce370b
chore(deps): Bump prost-types from 0.10.0 to 0.10.1 ( #4377 )
...
Bumps [prost-types](https://github.com/tokio-rs/prost ) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.10.0...v0.10.1 )
---
updated-dependencies:
- dependency-name: prost-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-21 13:22:53 +00:00
f444e63960
test: include materialized delete predicates in NG query tests ( #4371 )
...
* refactor: move `batch_filter` to `datafusion_util`
* fix: outdated docstring
* feat: allow passing record batches to `iox_tests` parquet files
* test: include materialized delete predicates in NG query tests
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-21 13:00:13 +00:00
b88900b08c
chore(deps): Bump nix from 0.23.1 to 0.24.0 ( #4378 )
...
Bumps [nix](https://github.com/nix-rust/nix ) from 0.23.1 to 0.24.0.
- [Release notes](https://github.com/nix-rust/nix/releases )
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md )
- [Commits](https://github.com/nix-rust/nix/commits )
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-21 12:51:31 +00:00
2984578aac
chore: Update pprof, remove old versions of prost ( #4374 )
...
* chore: Upgrade pprof to 0.8
* chore: Update heappy
2022-04-21 10:52:23 +00:00
200a066d46
chore(deps): Bump prost from 0.10.0 to 0.10.1 ( #4375 )
...
Bumps [prost](https://github.com/tokio-rs/prost ) from 0.10.0 to 0.10.1.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.10.0...v0.10.1 )
---
updated-dependencies:
- dependency-name: prost
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-21 10:25:10 +00:00
c084785bc3
feat: fuse ingester and catalog states in querier ( #4355 )
...
* feat: fuse ingester and catalog states in querier
This now correctly combines the data we get from the ingester w/ the
data we get from the catalog. Right now it bails out if during the very
small time windows between asking the ingester and querying the catalog
the compactor combines the newest files w/ "too new" files (see tests).
* fix: improve error wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* fix: improve doc comment
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* fix: explain tests
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: improve tests, method naming and docs
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 14:37:19 +00:00
3961568fff
fix: Add protobuf compiler to docker build image ( #4369 )
2022-04-20 13:33:06 +00:00
c05612935b
test: add end to end querier test with multiple ingesters ( #4309 )
...
* test: add a test with multiple ingesters
* docs: improve doc test
* refactor: Print out step number in StepTest
* fix: make timeout more explicit
* fix: add tests for merge_info
2022-04-20 11:56:41 +00:00
fcb82b2c99
Merge pull request #4367 from influxdata/dom/panic-metrics
...
feat: panic metrics
2022-04-20 11:43:20 +00:00
c363242902
refactor: emit panic metrics for server types
...
Configures the long-running / server modes to emit panic metrics.
2022-04-20 12:30:02 +01:00
49c38c2976
feat: emit thread panic count metric
...
Allows the panic handler to be optionally configured to emit a thread
panic metric, increasing by 1 each time a panic is observed.
2022-04-20 12:30:02 +01:00
73bed810da
chore: Update arrow, arrow-flight, parquet, tonic, prost, etc ( #4357 )
...
* chore: Update datafusion
* chore: Update arrow/arrow-flight/parquet to 12
* chore: update datafusion correctly
* chore: Update prost, tonic, and dependents
* fix: Fixup some api changes
* fix: Update test output in db
* fix: Update test output in parquet_file
* fix: remove old pbjson types
* fix: Add "--experimental_allow_proto3_optional" flag
* chore: Run cargo hakari tasks
* fix: compile error
* chore: Update heappy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-20 11:12:17 +00:00
Carol (Nichols || Goulding)
Nga Tran
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Jake Goulding
Carol (Nichols || Goulding)
Andrew Lamb
Marco Neumann