694ffd2238
chore(deps): Bump httparse from 1.6.0 to 1.7.0 ( #4277 )
...
Bumps [httparse](https://github.com/seanmonstar/httparse ) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/seanmonstar/httparse/releases )
- [Commits](https://github.com/seanmonstar/httparse/compare/v1.6.0...v1.7.0 )
---
updated-dependencies:
- dependency-name: httparse
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-12 09:37:08 +00:00
380cd9bbff
refactor: use a single flight client implementation ( #4273 )
...
"end-user -> querier" and "querier -> ingester" should use a single
Flight client implementation. The difference is just the request and
response metadata.
This changes our default Flight client to use protobuf instead of JSON
for the ticket format.
2022-04-12 09:08:25 +00:00
3f5eab7648
feat: allow the querier to talk with multiple ingesters ( #4271 )
...
* refactor: Move querier config to clap_blocks
* refactor: Add tests
* refactor: allow multiple addresses
* refactor: Update to use multiple addresses
* fix: bow to clippy
* fix: docstring
* fix: error if address is repeated multiple times
* chore: Add error enum, plumb through
* fix: clippy
* refactor: improve Rust API
* fix: fix test
2022-04-11 18:49:49 +00:00
d23d7b190f
fix: Filter compacted sort key to present columns
...
In parquet files written after compaction, use the catalog sort key but
filter it to only those columns that appear in the merged schema.
Panic if there are any columns in the merged schema's primary key that
aren't in the catalog sort key; that shouldn't happen.
2022-04-11 14:09:46 -04:00
48d3d0e471
fix: Panic earlier if a partition doesn't have a catalog sort key
...
Because we decided a panic was ok to do if the catalog doesn't have a
sort key for the partition, move the panic earlier to catch it before
doing other work.
2022-04-11 14:09:46 -04:00
5a46bb972d
refactor: Extract a helper fn for creating Arrays in RecordBatches
2022-04-11 14:09:46 -04:00
d75b161f44
feat: Add some logging when changing sort key in tests
2022-04-11 14:09:46 -04:00
628d54836a
refactor: Use a more specific function name
2022-04-11 14:09:46 -04:00
b6253b8046
docs: Explain why this panic might happen
2022-04-11 14:09:45 -04:00
55fe3b8d50
feat: Use the sort key stored in the catalog during compaction
...
Fixes #4249 .
2022-04-11 14:09:45 -04:00
87e4a1a51d
refactor: Move ingester sort key to schema sort key
...
This logic isn't actually ingester specific
2022-04-11 14:09:45 -04:00
a053077a05
refactor: Make compute_sort_key more general than the ingester
...
Enable computing sort keys for a schema and an iterator of record
batches.
2022-04-11 14:09:45 -04:00
941dcc8e80
fix: return error rather than panic in querier namspace access ( #4270 )
2022-04-11 14:01:15 +00:00
9bf15bfd78
Merge pull request #4269 from influxdata/dom/bump-rust
...
build: use rust 1.60
2022-04-11 11:58:22 +00:00
30fbbb14b8
Merge branch 'main' into dom/bump-rust
2022-04-11 11:50:40 +00:00
f6e6821276
feat: Add basic Querier <--> Ingester "Service Configuration" ( #4259 )
...
* feat: Add basic Querier <--> Ingester "Service Configuration"
* docs: update comments in test
* refactor: cleanup tests a little
* refactor: make trait more consistent
* docs: improve comments in IngesterPartition
2022-04-11 11:50:22 +00:00
3624d63529
build: use rust 1.60
2022-04-11 12:41:27 +01:00
b097b394b9
Merge pull request #4257 from influxdata/dom/ingester-sink-impl
...
feat(ingester): use SequencedStreamHandler
2022-04-11 10:59:57 +00:00
40d412502e
Merge branch 'main' into dom/ingester-sink-impl
2022-04-11 10:52:00 +00:00
be4ebe2563
feat: Add more context to error messages ( #4263 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-11 10:51:50 +00:00
d4a8fe30c2
Merge branch 'main' into dom/ingester-sink-impl
2022-04-11 11:47:49 +01:00
91991eaecf
chore(deps): Bump tracing from 0.1.32 to 0.1.33 ( #4268 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.32 to 0.1.33.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.32...tracing-0.1.33 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 08:35:55 +00:00
3689e17704
chore(deps): Bump croaring from 0.5.1 to 0.5.2 ( #4267 )
...
* chore(deps): Bump croaring from 0.5.1 to 0.5.2
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.5.1...0.5.2 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-11 08:22:43 +00:00
69e42bf478
chore(deps): Bump tracing-subscriber from 0.3.10 to 0.3.11 ( #4266 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.10 to 0.3.11.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.10...tracing-subscriber-0.3.11 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 08:12:30 +00:00
f838cb78a2
fix: not to add IOxReadFilterNode for empty non-duplicated chunks ( #4264 )
...
* fix: not to add IOxReadFilterNode for no data of non-duplicated chunks if there is already scan node for overlapped/duplicated chunks
* refactor: address review comments
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 21:03:22 +00:00
bbbdcc75a8
feat: `QuerierDatabase::chunks` returns `Result` ( #4260 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 18:54:17 +00:00
eb7d41f7a1
test: Add schema validation to end to end querier test ( #4258 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 18:11:00 +00:00
2cdd1951d9
fix: fix pprof ( #4261 )
2022-04-08 17:46:01 +00:00
5c3cbb14b4
test: join ingester background tasks
2022-04-08 14:24:56 +01:00
34e65c23fa
fix: Update for signature change ( #4252 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 11:21:07 +00:00
ffbb96f9fd
Merge pull request #4256 from influxdata/dom/compactor-path-debug
...
refactor: extra debug in compactor
2022-04-08 10:29:13 +00:00
dce939c580
refactor: use SequencedStreamHandler
...
Removes the old stream_in_sequenced_entries() write buffer handler,
replacing it with the SequencedStreamHandler introduced in #4203 .
This change will affect the metrics emitted by an ingester as outlined
in #4243 .
2022-04-08 11:28:39 +01:00
71a278ac7e
refactor: accept !Sync write buffer streams
...
Removes the Sync bound SequencedStreamHandler input stream type, as the
BoxStream returned by the WriteBufferStreamHandler is not Sync.
This change means the SequencedStreamHandler is not Sync either, but is
still Send and therefore can be moved into tokio tasks.
2022-04-08 11:28:39 +01:00
c2236fa3fb
feat: impl DmlSink for IngesterData
...
This commit adds an adaptor (IngestSinkAdaptor) that provides a DmlSink
implementation for the existing write path (IngesterData). With this,
the existing write path becomes compatible with the new
op stream handler (SequencedStreamHandler).
2022-04-08 11:28:39 +01:00
6131381b8d
refactor: extra debug in compactor
...
Continues pushing more debug through the compaction processing loop.
2022-04-08 11:20:19 +01:00
ba3eb409d9
chore(deps): Bump libc from 0.2.121 to 0.2.122 ( #4248 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.121 to 0.2.122.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.121...0.2.122 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:40:53 +00:00
db162980aa
chore(deps): Bump syn from 1.0.90 to 1.0.91 ( #4241 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.90 to 1.0.91.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.90...1.0.91 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:33:01 +00:00
a30a85e62c
feat: Add get_write_info service ( #4227 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:24:58 +00:00
a3ee11a8e8
Merge pull request #4243 from influxdata/dom/ingester-op-instrumentation
...
feat: ingester op instrumentation
2022-04-07 16:41:23 +00:00
8bd0bfb669
Merge branch 'main' into dom/ingester-op-instrumentation
2022-04-07 16:33:25 +00:00
0cd07c7a19
Merge pull request #4250 from influxdata/dom/compactor-path-debug
...
refactor: compactor debug logging
2022-04-07 16:28:28 +00:00
cce7329102
Merge branch 'main' into dom/compactor-path-debug
2022-04-07 16:20:33 +00:00
3706ac042d
refactor: add debug in compaction path
...
Adds debug!() and friends through the compaction path.
2022-04-07 17:13:45 +01:00
2607151ec9
refactor: print parquet path in Debug impl
...
Print the actual path being used when debug-printing a ParquetFilePath.
2022-04-07 16:22:43 +01:00
e6c0780987
Merge pull request #4228 from influxdata/cn/sort-key-across-persists
...
feat: Use and update partition sort_key in the catalog
2022-04-07 14:49:17 +00:00
f5996c5ab4
Merge branch 'main' into cn/sort-key-across-persists
2022-04-07 14:40:55 +00:00
b196731892
Merge branch 'main' into dom/ingester-op-instrumentation
2022-04-07 12:22:58 +01:00
998a66fd98
docs: Update ingester/src/stream_handler/sink_instrumentation.rs
...
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-07 12:18:14 +01:00
30c3ef5aa6
fix: Only save relevant columns in parquet file's sort key
2022-04-06 14:09:08 -04:00
1de280e60f
chore: Update datafusion ( #4247 )
2022-04-06 16:22:44 +00:00
dependabot[bot]
Marco Neumann
Andrew Lamb
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
kodiakhq[bot]
Dom Dwyer
Nga Tran