a053077a05
refactor: Make compute_sort_key more general than the ingester
...
Enable computing sort keys for a schema and an iterator of record
batches.
2022-04-11 14:09:45 -04:00
941dcc8e80
fix: return error rather than panic in querier namspace access ( #4270 )
2022-04-11 14:01:15 +00:00
9bf15bfd78
Merge pull request #4269 from influxdata/dom/bump-rust
...
build: use rust 1.60
2022-04-11 11:58:22 +00:00
30fbbb14b8
Merge branch 'main' into dom/bump-rust
2022-04-11 11:50:40 +00:00
f6e6821276
feat: Add basic Querier <--> Ingester "Service Configuration" ( #4259 )
...
* feat: Add basic Querier <--> Ingester "Service Configuration"
* docs: update comments in test
* refactor: cleanup tests a little
* refactor: make trait more consistent
* docs: improve comments in IngesterPartition
2022-04-11 11:50:22 +00:00
3624d63529
build: use rust 1.60
2022-04-11 12:41:27 +01:00
b097b394b9
Merge pull request #4257 from influxdata/dom/ingester-sink-impl
...
feat(ingester): use SequencedStreamHandler
2022-04-11 10:59:57 +00:00
40d412502e
Merge branch 'main' into dom/ingester-sink-impl
2022-04-11 10:52:00 +00:00
be4ebe2563
feat: Add more context to error messages ( #4263 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-11 10:51:50 +00:00
d4a8fe30c2
Merge branch 'main' into dom/ingester-sink-impl
2022-04-11 11:47:49 +01:00
91991eaecf
chore(deps): Bump tracing from 0.1.32 to 0.1.33 ( #4268 )
...
Bumps [tracing](https://github.com/tokio-rs/tracing ) from 0.1.32 to 0.1.33.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.32...tracing-0.1.33 )
---
updated-dependencies:
- dependency-name: tracing
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 08:35:55 +00:00
3689e17704
chore(deps): Bump croaring from 0.5.1 to 0.5.2 ( #4267 )
...
* chore(deps): Bump croaring from 0.5.1 to 0.5.2
Bumps [croaring](https://github.com/saulius/croaring-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/saulius/croaring-rs/releases )
- [Commits](https://github.com/saulius/croaring-rs/compare/0.5.1...0.5.2 )
---
updated-dependencies:
- dependency-name: croaring
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-11 08:22:43 +00:00
69e42bf478
chore(deps): Bump tracing-subscriber from 0.3.10 to 0.3.11 ( #4266 )
...
Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing ) from 0.3.10 to 0.3.11.
- [Release notes](https://github.com/tokio-rs/tracing/releases )
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.3.10...tracing-subscriber-0.3.11 )
---
updated-dependencies:
- dependency-name: tracing-subscriber
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 08:12:30 +00:00
f838cb78a2
fix: not to add IOxReadFilterNode for empty non-duplicated chunks ( #4264 )
...
* fix: not to add IOxReadFilterNode for no data of non-duplicated chunks if there is already scan node for overlapped/duplicated chunks
* refactor: address review comments
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 21:03:22 +00:00
bbbdcc75a8
feat: `QuerierDatabase::chunks` returns `Result` ( #4260 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 18:54:17 +00:00
eb7d41f7a1
test: Add schema validation to end to end querier test ( #4258 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 18:11:00 +00:00
2cdd1951d9
fix: fix pprof ( #4261 )
2022-04-08 17:46:01 +00:00
5c3cbb14b4
test: join ingester background tasks
2022-04-08 14:24:56 +01:00
34e65c23fa
fix: Update for signature change ( #4252 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-08 11:21:07 +00:00
ffbb96f9fd
Merge pull request #4256 from influxdata/dom/compactor-path-debug
...
refactor: extra debug in compactor
2022-04-08 10:29:13 +00:00
dce939c580
refactor: use SequencedStreamHandler
...
Removes the old stream_in_sequenced_entries() write buffer handler,
replacing it with the SequencedStreamHandler introduced in #4203 .
This change will affect the metrics emitted by an ingester as outlined
in #4243 .
2022-04-08 11:28:39 +01:00
71a278ac7e
refactor: accept !Sync write buffer streams
...
Removes the Sync bound SequencedStreamHandler input stream type, as the
BoxStream returned by the WriteBufferStreamHandler is not Sync.
This change means the SequencedStreamHandler is not Sync either, but is
still Send and therefore can be moved into tokio tasks.
2022-04-08 11:28:39 +01:00
c2236fa3fb
feat: impl DmlSink for IngesterData
...
This commit adds an adaptor (IngestSinkAdaptor) that provides a DmlSink
implementation for the existing write path (IngesterData). With this,
the existing write path becomes compatible with the new
op stream handler (SequencedStreamHandler).
2022-04-08 11:28:39 +01:00
6131381b8d
refactor: extra debug in compactor
...
Continues pushing more debug through the compaction processing loop.
2022-04-08 11:20:19 +01:00
ba3eb409d9
chore(deps): Bump libc from 0.2.121 to 0.2.122 ( #4248 )
...
Bumps [libc](https://github.com/rust-lang/libc ) from 0.2.121 to 0.2.122.
- [Release notes](https://github.com/rust-lang/libc/releases )
- [Commits](https://github.com/rust-lang/libc/compare/0.2.121...0.2.122 )
---
updated-dependencies:
- dependency-name: libc
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:40:53 +00:00
db162980aa
chore(deps): Bump syn from 1.0.90 to 1.0.91 ( #4241 )
...
Bumps [syn](https://github.com/dtolnay/syn ) from 1.0.90 to 1.0.91.
- [Release notes](https://github.com/dtolnay/syn/releases )
- [Commits](https://github.com/dtolnay/syn/compare/1.0.90...1.0.91 )
---
updated-dependencies:
- dependency-name: syn
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:33:01 +00:00
a30a85e62c
feat: Add get_write_info service ( #4227 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-07 19:24:58 +00:00
a3ee11a8e8
Merge pull request #4243 from influxdata/dom/ingester-op-instrumentation
...
feat: ingester op instrumentation
2022-04-07 16:41:23 +00:00
8bd0bfb669
Merge branch 'main' into dom/ingester-op-instrumentation
2022-04-07 16:33:25 +00:00
0cd07c7a19
Merge pull request #4250 from influxdata/dom/compactor-path-debug
...
refactor: compactor debug logging
2022-04-07 16:28:28 +00:00
cce7329102
Merge branch 'main' into dom/compactor-path-debug
2022-04-07 16:20:33 +00:00
3706ac042d
refactor: add debug in compaction path
...
Adds debug!() and friends through the compaction path.
2022-04-07 17:13:45 +01:00
2607151ec9
refactor: print parquet path in Debug impl
...
Print the actual path being used when debug-printing a ParquetFilePath.
2022-04-07 16:22:43 +01:00
e6c0780987
Merge pull request #4228 from influxdata/cn/sort-key-across-persists
...
feat: Use and update partition sort_key in the catalog
2022-04-07 14:49:17 +00:00
f5996c5ab4
Merge branch 'main' into cn/sort-key-across-persists
2022-04-07 14:40:55 +00:00
b196731892
Merge branch 'main' into dom/ingester-op-instrumentation
2022-04-07 12:22:58 +01:00
998a66fd98
docs: Update ingester/src/stream_handler/sink_instrumentation.rs
...
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-07 12:18:14 +01:00
30c3ef5aa6
fix: Only save relevant columns in parquet file's sort key
2022-04-06 14:09:08 -04:00
1de280e60f
chore: Update datafusion ( #4247 )
2022-04-06 16:22:44 +00:00
24eeddce8a
chore: fix lint warnings
2022-04-06 16:45:31 +01:00
091640bb23
feat: emit tracing span for op apply
...
This commit uses the tracing metadata within the DmlOperation to emit a
tracing span from the ingester covering the DmlSink::apply() operation.
2022-04-06 16:32:00 +01:00
f6c65f52a3
refactor: impl WatermarkFetcher
...
Implement WatermarkFetcher for PeriodicWatermarkFetcher and remove
unnecessary async.
2022-04-06 16:32:00 +01:00
436da19d9a
feat: DmlSink instrumentation
...
This commit adds the SinkInstrumentation type that decorates an inner
DmlSink with call latency and write buffer metrics.
The write buffer / sink call metrics may be split apart into two
separate responsibilities in the future if there are multiple DmlSink
that need instrumentation, but deferring adding more types until it is
needed.
2022-04-06 16:32:00 +01:00
c244b03281
feat: Add `SequencerProgress` reporting to ingester ( #4238 )
...
* feat: Add `SequencerProgress` reporting to ingester
* refactor: Use KafkaPartition in write_summary
* fix: Update docstrings
* refactor: Change ingester to use KafkaPartition everywhere
* refactor: add SequencerProgress::combine
* refactor: return new SequencerProgress rather than updating
* fix: distinguish between yes/no/unknown in WriteSummary
* docs: Update data_types2/src/lib.rs
Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: Paul Dix <paul@pauldix.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-04-06 15:13:21 +00:00
438e739344
chore(deps): Bump parquet from 11.0.0 to 11.1.0 ( #4240 )
...
* chore(deps): Bump parquet from 11.0.0 to 11.1.0
Bumps [parquet](https://github.com/apache/arrow-rs ) from 11.0.0 to 11.1.0.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/apache/arrow-rs/compare/11.0.0...11.1.0 )
---
updated-dependencies:
- dependency-name: parquet
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: Update tests
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-04-06 14:51:01 +00:00
bf3cb45723
refactor: Pass PartitionInfo as argument
2022-04-06 09:31:42 -04:00
f0d5987317
feat: Update partition sort_key in catalog after persist
...
Connects to #4196 .
2022-04-06 09:31:42 -04:00
c11fe5b226
refactor: Use the new contains method on SortKey
2022-04-06 09:31:42 -04:00
b16fcc284d
feat: Add new columns to the sort key during compaction
...
Connects to #4196 .
2022-04-06 09:31:42 -04:00
98d052dba7
feat: Use catalog sort key if specified
...
Pass the sort key from the catalog through to compact_persisting_batch.
If the sort key is Some, use that. If the sort key is None, compute it
from the data's cardinality with compute_sort_key.
Connects to #4196 .
2022-04-06 09:31:42 -04:00
Carol (Nichols || Goulding)
Andrew Lamb
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Nga Tran