1ad0160d30
chore(deps): Bump serde_json from 1.0.94 to 1.0.95 ( #7348 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.94 to 1.0.95.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.94...v1.0.95 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-28 11:09:59 +00:00
5a965e2481
Merge branch 'main' into savage/add-tests-for-hot-partition-persistence
2023-03-28 11:37:20 +01:00
c7b3565990
refactor: Compare data inside the partition as persisted
...
Use tokio::task::yield_now to allow the persist task to
be enqueued instead of tokio::time::advance.
2023-03-28 11:28:53 +01:00
621672b38f
test: Add an e2e test for write replication and fix it because it didn't work 😜 ( #7344 )
...
* test: Add an e2e test for write replication
* fix: Pass through rpc_write_replicas configuration to RpcWrite handler
---------
Co-authored-by: Dom <dom@itsallbroken.com>
2023-03-28 09:31:16 +00:00
f588bd5e8d
chore: update `spin` to 0.9.7 ( #7350 )
...
0.9.6 was yanked.
2023-03-28 08:47:25 +00:00
0db347bc1a
Merge remote-tracking branch 'origin/main' into jrb_21_backfill_handling
...
# Conflicts:
# compactor2/tests/layouts/knobs.rs
# compactor2/tests/layouts/large_files.rs
# compactor2/tests/layouts/many_files.rs
2023-03-27 14:20:16 -05:00
638a8ea4a8
chore: move random backfill tests to their own file
2023-03-27 13:48:16 -05:00
ed282ed738
chore: add case to cause non-overlapped file to be processed by high_l0_overlap_split
2023-03-27 13:40:06 -05:00
6a60914f95
chore: insta updates
2023-03-27 11:15:24 -05:00
af71e3295c
Merge pull request #7342 from influxdata/dom/cleanup-mock-iter
...
refactor(test): generic return iterator
2023-03-27 15:01:25 +01:00
de043f93e5
feat(ingester2): Add test coverage for hot partition persistence
...
This commit adds a test for the hot partition persistence
post-write observer functionality.
2023-03-27 12:46:54 +01:00
874325d9ce
refactor(test): generic return iterator
...
This change allows the MockWriteClient to accept more input types and
internalises the box dyn bits; I got tired of writing "Box::new()"
everywhere.
2023-03-27 13:31:31 +02:00
71b88b22b9
fix: ensure we don't loose predicates in chunk roundtrips ( #7340 )
...
`extract_chunks` never runs after predicate pushdown. However IF this
should ever happen, we would potentially forget the predicates attached
to `ParquetExec`. So let's make sure we refuse chunk extraction in this
case. This is similar to the existing behavior, i.e. we don't support
chunk extraction after filter pushdown (i.e. if there is a filter around
an `RecordBatchesExec`).
For #6098 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-27 11:18:56 +00:00
44551e7519
chore(deps): Bump regex from 1.7.2 to 1.7.3 ( #7338 )
...
Bumps [regex](https://github.com/rust-lang/regex ) from 1.7.2 to 1.7.3.
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.7.2...1.7.3 )
---
updated-dependencies:
- dependency-name: regex
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-27 09:20:35 +00:00
e067cfd3d7
Merge pull request #7316 from influxdata/dom/replicate
...
feat: write replication
2023-03-27 10:13:11 +01:00
48333f66a4
Merge branch 'main' into dom/replicate
2023-03-27 10:06:07 +01:00
e24fc9d2d8
Merge pull request #7339 from influxdata/dependabot/cargo/mockito-1.0.1
...
chore(deps): Bump mockito from 1.0.0 to 1.0.1
2023-03-27 10:05:18 +01:00
b62099ea4a
Merge branch 'main' into dom/replicate
2023-03-27 10:04:54 +01:00
6105fd5bb7
refactor: compute n_copies once
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-03-27 10:04:17 +01:00
6a4d55e905
chore(deps): Bump mockito from 1.0.0 to 1.0.1
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/1.0.0...1.0.1 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-27 08:35:31 +00:00
00a6676475
chore(deps): Bump predicates from 3.0.1 to 3.0.2 ( #7337 )
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v3.0.1...v3.0.2 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 08:32:27 +00:00
010d35c2b5
chore(deps): Bump indexmap from 1.9.2 to 1.9.3 ( #7336 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.9.2 to 1.9.3.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md )
- [Commits](https://github.com/bluss/indexmap/compare/1.9.2...1.9.3 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 08:19:43 +00:00
976dadb4d4
chore: split out the time range selection for splitting into separate function
2023-03-24 16:59:02 -05:00
0d66b04f68
refactor(flightsql): Build `GetTables` directly from DataFusion catalog ( #7325 )
...
* refactor: Build GetTables response directly from SessionContext
* fix: fmt
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 19:15:24 +00:00
f73187ff7e
feat: add interpolation fill strategy to GapFillExec ( #7317 )
...
* feat: add interpolation fill strategy to GapFillExec
* chore: clippy
* chore: code review feedback
* chore: fix doc comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:53:14 +00:00
1817104236
chore: Update DataFusion again ( #7329 )
...
* chore: update datafusion
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:45:53 +00:00
35865e1ab3
refactor(flightsql): Build `GetCatalogs` directly from DataFusion catalog ( #7327 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:34:56 +00:00
4b888c7255
chore(deps): Bump insta from 1.28.0 to 1.29.0 ( #7322 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.28.0 to 1.29.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.28.0...1.29.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:25:01 +00:00
dcf32fa1d7
Merge pull request #7319 from influxdata/cn/simulator-rules
...
feat: Record why compact, split, or nothing plans were picked
2023-03-24 17:55:44 +00:00
60e2cb0e2b
Merge branch 'main' into cn/simulator-rules
2023-03-24 17:45:43 +00:00
07d5011d7a
refactor: Rename TriedToCompactTooMuch to CompactAndSplitOutput
2023-03-24 12:05:15 -04:00
5dd08a233d
refactor: Rename FoundFilesToSplit to ReduceLargeFileSize
2023-03-24 11:59:52 -04:00
fc1bea8502
docs: Add comments when the reason chosen is irrelevant to the test
2023-03-24 11:56:40 -04:00
ce77d3bd74
refactor(flightsql): Build `GetTableTypes` response directly from the catalog ( #7324 )
...
* refactor(flightsql): Do not use plan for GetTableTypes
* test: Add an end to end test
2023-03-24 15:20:28 +00:00
18609381f0
chore: Update datafusion ( #7318 )
...
* chore: Update datafusion
* chore: Update for API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 15:09:36 +00:00
5dd71998a1
chore: Update datafusion ( #7318 )
...
* chore: Update datafusion
* chore: Update for API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 15:07:23 +00:00
3eb5c916d9
chore: insta updates
2023-03-24 10:01:37 -05:00
5a49ae0efd
feat: vertical split files when L0s are highly overlapped
2023-03-24 10:01:07 -05:00
9cd2a8816a
feat(flightsql): Support `include_schema` param in `GetTables` metadata endpoint ( #7307 )
...
* feat(flightsql): Add support for table_schema in GetTables and complies
feat(fightsql): Add support for table_schema in GetTables
support actual schema
it compiles / does not fail
* chore: resolve merge conflict
* chore: make table_schema optional
* test: update e2e test for `include_schema` = true
* chore: remove info!() and update test `flightsql_schema_matches`
* chore(deps): Bump rustix from 0.36.11 to 0.37.3 (#7308 )
* chore(deps): Bump rustix from 0.36.11 to 0.37.3
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.11 to 0.37.3.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.11...v0.37.3 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* chore: make real error for existing columns
* chore: user match instead of unwrap() on column names
* chore: use datafusion::physical_plan::collect() to get record batches
* chore: use `concat_batches` to combine multiple batches into single one and fix db schema test
* chore: add doc comment for GetTables
* chore: remove pretty print
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 14:59:30 +00:00
3fe8bda51a
feat: Record why compact, split, or nothing plans were picked
...
And display the reason in the simulator tests to ensure decisions are
consistent.
Fixes influxdata/idpe#17306 .
2023-03-23 16:58:17 -04:00
c82674e88d
fix: Parse empty single-line comments ( #7312 )
2023-03-23 20:22:49 +00:00
99b5e0b6ec
feat: add version and uuid to startup message ( #7264 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 16:55:17 +00:00
feae0ce345
feat(router): configurable replication factor
...
This change allows the replication factor to be specified at runtime,
defaulting to "replication disabled".
2023-03-23 17:48:42 +01:00
125fef388c
feat: MVP replication support
...
This commit implements replication for the router's RpcWrite handler.
The desired number of replica copies is specified at startup time, and
each user write will be fanned-out with the specified replication factor
(replicas + 1).
A failure to write to any upstreams returns the write error, but a
failure to obtain enough ACKs (enough successful writes) after at least
1 ACK will return a "partial write" error - this differentiation is
important, as the user's write will be readable after a partial write
error has occurred.
This currently writes to upstreams serially; this is clearly an
opportunity for improvement! A follow-on PR will parallelise writes
across the desired number of replicas while maintaining the "at most one
ack'd write to one host" invariant.
Note that replication is currently hard-coded as disabled.
2023-03-23 17:48:41 +01:00
520140c322
Merge pull request #7315 from influxdata/dom/rpc-timeout-error
...
feat(router): preserve RPC error response in user response
2023-03-23 16:47:39 +00:00
53a3264d14
Merge branch 'main' into dom/rpc-timeout-error
2023-03-23 16:07:04 +00:00
1296eab093
Merge pull request #7306 from influxdata/cn/refactor
...
refactor: Organize compactor component responsibilities
2023-03-23 15:34:29 +00:00
acb491aa98
Merge branch 'main' into cn/refactor
2023-03-23 15:27:46 +00:00
2379f00929
test: set max_l0_created_at appropriately for tests of many_files ( #7304 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 15:27:28 +00:00
98ec67c564
docs: Add suggested clarification comment
2023-03-23 11:20:06 -04:00
dependabot[bot]
Fraser Savage
Carol (Nichols || Goulding)
Marco Neumann
Joe-Blount
Dom
Andrew Lamb
Christopher M. Wolff
kodiakhq[bot]
Carol (Nichols || Goulding)
Chunchun Ye
Stuart Carnie
Nga Tran