d0db73d8de
chore(deps): Bump clap from 3.2.15 to 3.2.16 ( #5239 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.15 to 3.2.16.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.15...v3.2.16 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-01 08:00:47 +00:00
a687844ddf
chore(deps): Bump bytes from 1.2.0 to 1.2.1 ( #5244 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.2.0 to 1.2.1.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/commits )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-01 07:50:06 +00:00
7cc8486e5a
fix: remove left over `deb!` macro ( #5224 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-30 15:33:02 +00:00
0e9695f202
feat: add a few helpful compactor debug logs ( #5235 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 17:38:33 +00:00
87bdabb38a
feat: log external span for query gRPC requests ( #5187 )
...
* feat: log external span for query gRPC requests
This should simplify the correlation with our binlog data.
* refactor: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 12:53:12 +00:00
3fbe860bb9
fix: interpret [MIN_NANO_TIME, MAX_NANO_TIME) range as all time for optimization ( #5231 )
...
InfluxQL queries can send (technically incorrect) ranges like this, meaning all time
but excluding the max nanosecond time.
Since this is an important case, we should handle it specially and use the optimized
'all time' handling for meta queries even though this is technically wrong in that
it does not filter out column names / measurement names at MAX_NANO_TIME exactly.
Closes: https://github.com/influxdata/conductor/issues/1072
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 12:24:26 +00:00
9215a534d0
chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0` ( #5229 )
...
* chore: Update datafusion and `arrow`/`parquet`/`arrow-flight` to `19.0.0`
* chore: Run cargo hakari tasks
* fix: Update for API changes
* fix: clippy
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-28 08:10:47 +00:00
fcce00bf09
feat: run many compact partitions in parallel ( #5230 )
...
* feat: run many compact partitions in parallel
* refactor: Use rust futures fu to run compactor jobs in parallel
* chore: Apply suggestions from code review
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-27 20:55:45 +00:00
7eebe061a6
fix: reduce log verbosity for `found compaction candidates` message ( #5225 )
...
* fix: reduce log verbosity
* refactor: sleep for a sec if no work, print debug
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-27 19:35:31 +00:00
9da8062a16
fix: Fix typo in log message ( #5222 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-27 15:34:37 +00:00
9a9a1a4777
feat: limit per-table chunk data for every query ( #5223 )
...
* feat: `QueryChunk::as_any`
* feat: allo `ChunkPruner::prune_chunks` to fail
* feat: limit per-table chunk data for every query
Closes #5211 .
* fix: address review comments
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-27 13:20:05 +00:00
6d01a9ad68
fix: Clarify error msg: line number is 1-based ( #5219 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-27 08:37:28 +00:00
d7ab7362fd
refactor: avoid schema copies in `select_schema` ( #5214 )
...
This massively helps with #5202 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-27 08:30:26 +00:00
85c186f5b8
feat: cache projected chunk schemas in querier ( #5213 )
...
* feat: cache projected chunk schemas in querier
Ref #5202 .
* refactor: simplify size calculations
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-27 08:23:20 +00:00
2bbc419f95
fix: Tell which column failed typecheck ( #5220 )
2022-07-27 08:16:15 +00:00
de22b6b080
test: Add a test for LP quoting ( #5210 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-26 20:08:37 +00:00
727f0152d0
chore(deps): Bump tokio from 1.20.0 to 1.20.1 ( #5209 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.20.0 to 1.20.1.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.20.0...tokio-1.20.1 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-26 15:36:25 +00:00
bbcf4ec64e
fix: Run compactor streams in parallel to avoid deadlock ( #5212 )
...
* fix: run compaction streams at once
* fix: make it compile
* fix: improve wording
* fix: use task to write parquet files in parallel
2022-07-26 12:17:38 +00:00
7d16ac7de0
docs: extend profiling guide ( #5205 )
...
* docs: extend profiling guide
More tools.
* chore: fix docs lint for `localhost` links
* docs: do not duplicate tracing docs
* refactor: clean up `lint_docs` and strip anchors from relative links
2022-07-26 09:40:42 +00:00
246b294b05
chore(deps): Bump clap from 3.2.14 to 3.2.15 ( #5208 )
...
Bumps [clap](https://github.com/clap-rs/clap ) from 3.2.14 to 3.2.15.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/v3.2.15/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v3.2.14...v3.2.15 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-26 08:44:02 +00:00
d05f383a98
refactor: reduce compacting size and compacted file size to prevent compactor from waiting for reading a large file forever ( #5206 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 20:08:11 +00:00
5f15d97dd1
chore: always pass `ROARING_ARCH` ( #5203 )
...
* chore: always pass `ROARING_ARCH`
Always pass the `ROARING_ARCH` that we would use for our prod builds.
Otherwise this can easily be missed during testing, profiling or build
system changes (e.g. should we ever move aways for our `Dockerfile`).
This feature was introduced with Rust/Cargo 1.56.
* docs: explain env passing
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
2022-07-25 16:15:39 +00:00
614ca5ca96
refactor: allocate less memory for tracing ( #5200 )
...
While I could not find evidence that these allocations are a problem,
the metadata and links of spans are rarely used so we shouldn't pay for
them even for heavily traced applications.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:52:29 +00:00
e4dc8c2067
refactor: rename garbage collector crates for consistency ( #5196 )
...
* refactor: rename garbage collector crates for consistency
* fix: cargo fmt
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-25 12:44:37 +00:00
96c3a05481
feat: add debug log to `with_clear_timestamp_if_max_range` ( #5199 )
2022-07-25 11:43:54 +00:00
66af2bdd88
refactor: Split up `delete_three_delete_three_chunks.sql` test case ( #5197 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 20:57:31 +00:00
f4d0f13689
feat: split large compactions ( #5195 )
...
* feat: Split large compactions into multiple compacted files
Connects to #5121
* refactor: Extract update catalog function and error type
* refactor: Share physical plan to object store streaming
And only differ in the logical plan building based on split times in
different compaction cases.
* fix: Test for a split time equal to the max time and don't split then
2022-07-22 20:35:31 +00:00
69640c0ba5
feat: Different branch to hook up new compaction algorithm ( #5194 )
...
* chore: cherry pick the first 3 commits of branch cn/connect-new-compaction
* fix: modify the test to work correctly with compactor running
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 19:29:47 +00:00
465a69c41d
chore: Update datafusion ( #5193 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 18:07:38 +00:00
94343b1f27
fix: compute_split_time returns one value when min_time = max_time ( #5192 )
...
* test: Document the behavior of compute_split_time when min time = max time
* fix: compute_split_time returns one value when min_time = max_time
Co-authored-by: NGA-TRAN <nga-tran@live.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 17:29:50 +00:00
bbe07fcc79
feat: metrics for selection partition candidates for compaction ( #5190 )
...
* feat: metrics for selection partition candidates for compaction
* chore: Apply suggestions from code review
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
* refactor: remove unused metric labels
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 15:25:53 +00:00
c5451afa08
feat: Implement LP builder ( #5183 )
...
Helps with https://github.com/influxdata/influxrpc_compare/issues/179
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 14:41:10 +00:00
db9da93d40
Merge pull request #5189 from influxdata/cn/remove-file-num-restriction
...
fix: Compact one level 0 file to avoid getting stuck on it
2022-07-22 14:00:33 +00:00
d9ca28e83a
fix: Compact one level 0 file to avoid getting stuck on it
...
If compaction is called on one level 0 file, the work of compaction
doesn't really need to be done, but for simplicity's sake for now, run
it through the compaction query/write process rather than special casing
it. This will keep us from getting stuck in the unlikely event that a
partition with one level 0 file gets selected as a top candidate for
compaction.
2022-07-22 09:17:57 -04:00
fbf672015e
refactor: Reduce ceremony requried to create a `Span` from `SpanContext` ( #5181 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 11:19:38 +00:00
495bbe48f2
refactor: Reduce boiler plate calling `SpanRecorder::child` ( #5180 )
...
* refactor: call SpanRecorder::child
* refactor: update more locations
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-22 11:11:45 +00:00
5a0af921c8
chore: Roll forward: Sync ReadWindowAggregate API: TagKeyMetaNames ( #5186 )
...
This reverts commit 5d02c755687ef041f5f45dbfc3e633a833284edb.
2022-07-22 10:44:06 +00:00
baee020efe
refactor: improve query gRPC logging ( #5185 )
...
- ensure that logging is done BEFORE the DB/namespace is requested (i.e.
any actual work is done) but AFTER the query semaphore is acquired
- simplify tag key decoding (so that the logging statements are simpler
to write)
2022-07-22 10:30:45 +00:00
07cdb99192
chore: Revert "Sync ReadWindowAggregate API: TagKeyMetaNames" ( #5184 )
...
We're noticing a possible regression (OOMs) in our testing cluster that roughly correlates with this.
2022-07-22 09:26:42 +00:00
e6d68f90b4
chore(deps): Bump permutation from 0.4.0 to 0.4.1 ( #5182 )
...
Bumps [permutation](https://github.com/jeremysalwen/rust-permutations ) from 0.4.0 to 0.4.1.
- [Release notes](https://github.com/jeremysalwen/rust-permutations/releases )
- [Commits](https://github.com/jeremysalwen/rust-permutations/commits )
---
updated-dependencies:
- dependency-name: permutation
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-07-22 08:38:23 +00:00
a2307648f5
Merge pull request #5176 from influxdata/cn/compact-filtered-files
...
feat: Compact filtered parquet files
2022-07-21 21:00:37 +00:00
0af73acfa2
Merge branch 'main' into cn/compact-filtered-files
2022-07-21 20:52:40 +00:00
a2c82a6f1c
chore: remove min sequence number from the catalog table as we no longer use it ( #5178 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-07-21 20:47:55 +00:00
ef70dd40d6
Merge pull request #5173 from influxdata/crepererum/issue5129j
...
feat: wire up cache tracing (2/2)
2022-07-21 20:35:10 +00:00
fce281a034
Merge branch 'main' into crepererum/issue5129j
2022-07-21 20:27:16 +00:00
aa030ba132
test: Additional coverage for the compaction operation
2022-07-21 16:01:26 -04:00
86c50b8033
fix: Use 0 for level 1 chunk order and max seq num for level 0 chunk order
2022-07-21 15:49:52 -04:00
f847365b1a
fix: Use actual partition values rather than placeholders
2022-07-21 15:10:24 -04:00
d46ec31aa1
feat: Compact filtered parquet files
...
Connects to #5121 .
2022-07-21 13:37:36 -04:00
50186ef5ee
feat: add sort key and partition key into PartitionCompactionCandidateWithInfo ( #5175 )
2022-07-21 16:57:17 +00:00
dependabot[bot]
Andrew Lamb
Marco Neumann
Sam Arnold
Nga Tran
Marko Mikulicic
Carol (Nichols || Goulding)
kodiakhq[bot]
Carol (Nichols || Goulding)