e067cfd3d7
Merge pull request #7316 from influxdata/dom/replicate
...
feat: write replication
2023-03-27 10:13:11 +01:00
48333f66a4
Merge branch 'main' into dom/replicate
2023-03-27 10:06:07 +01:00
e24fc9d2d8
Merge pull request #7339 from influxdata/dependabot/cargo/mockito-1.0.1
...
chore(deps): Bump mockito from 1.0.0 to 1.0.1
2023-03-27 10:05:18 +01:00
b62099ea4a
Merge branch 'main' into dom/replicate
2023-03-27 10:04:54 +01:00
6105fd5bb7
refactor: compute n_copies once
...
Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
2023-03-27 10:04:17 +01:00
6a4d55e905
chore(deps): Bump mockito from 1.0.0 to 1.0.1
...
Bumps [mockito](https://github.com/lipanski/mockito ) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/lipanski/mockito/releases )
- [Commits](https://github.com/lipanski/mockito/compare/1.0.0...1.0.1 )
---
updated-dependencies:
- dependency-name: mockito
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-03-27 08:35:31 +00:00
00a6676475
chore(deps): Bump predicates from 3.0.1 to 3.0.2 ( #7337 )
...
Bumps [predicates](https://github.com/assert-rs/predicates-rs ) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases )
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v3.0.1...v3.0.2 )
---
updated-dependencies:
- dependency-name: predicates
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 08:32:27 +00:00
010d35c2b5
chore(deps): Bump indexmap from 1.9.2 to 1.9.3 ( #7336 )
...
Bumps [indexmap](https://github.com/bluss/indexmap ) from 1.9.2 to 1.9.3.
- [Release notes](https://github.com/bluss/indexmap/releases )
- [Changelog](https://github.com/bluss/indexmap/blob/master/RELEASES.md )
- [Commits](https://github.com/bluss/indexmap/compare/1.9.2...1.9.3 )
---
updated-dependencies:
- dependency-name: indexmap
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-27 08:19:43 +00:00
976dadb4d4
chore: split out the time range selection for splitting into separate function
2023-03-24 16:59:02 -05:00
0d66b04f68
refactor(flightsql): Build `GetTables` directly from DataFusion catalog ( #7325 )
...
* refactor: Build GetTables response directly from SessionContext
* fix: fmt
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 19:15:24 +00:00
f73187ff7e
feat: add interpolation fill strategy to GapFillExec ( #7317 )
...
* feat: add interpolation fill strategy to GapFillExec
* chore: clippy
* chore: code review feedback
* chore: fix doc comments
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:53:14 +00:00
1817104236
chore: Update DataFusion again ( #7329 )
...
* chore: update datafusion
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:45:53 +00:00
35865e1ab3
refactor(flightsql): Build `GetCatalogs` directly from DataFusion catalog ( #7327 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:34:56 +00:00
4b888c7255
chore(deps): Bump insta from 1.28.0 to 1.29.0 ( #7322 )
...
Bumps [insta](https://github.com/mitsuhiko/insta ) from 1.28.0 to 1.29.0.
- [Release notes](https://github.com/mitsuhiko/insta/releases )
- [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md )
- [Commits](https://github.com/mitsuhiko/insta/compare/1.28.0...1.29.0 )
---
updated-dependencies:
- dependency-name: insta
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 18:25:01 +00:00
dcf32fa1d7
Merge pull request #7319 from influxdata/cn/simulator-rules
...
feat: Record why compact, split, or nothing plans were picked
2023-03-24 17:55:44 +00:00
60e2cb0e2b
Merge branch 'main' into cn/simulator-rules
2023-03-24 17:45:43 +00:00
07d5011d7a
refactor: Rename TriedToCompactTooMuch to CompactAndSplitOutput
2023-03-24 12:05:15 -04:00
5dd08a233d
refactor: Rename FoundFilesToSplit to ReduceLargeFileSize
2023-03-24 11:59:52 -04:00
fc1bea8502
docs: Add comments when the reason chosen is irrelevant to the test
2023-03-24 11:56:40 -04:00
ce77d3bd74
refactor(flightsql): Build `GetTableTypes` response directly from the catalog ( #7324 )
...
* refactor(flightsql): Do not use plan for GetTableTypes
* test: Add an end to end test
2023-03-24 15:20:28 +00:00
18609381f0
chore: Update datafusion ( #7318 )
...
* chore: Update datafusion
* chore: Update for API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 15:09:36 +00:00
5dd71998a1
chore: Update datafusion ( #7318 )
...
* chore: Update datafusion
* chore: Update for API change
* chore: Run cargo hakari tasks
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 15:07:23 +00:00
3eb5c916d9
chore: insta updates
2023-03-24 10:01:37 -05:00
5a49ae0efd
feat: vertical split files when L0s are highly overlapped
2023-03-24 10:01:07 -05:00
9cd2a8816a
feat(flightsql): Support `include_schema` param in `GetTables` metadata endpoint ( #7307 )
...
* feat(flightsql): Add support for table_schema in GetTables and complies
feat(fightsql): Add support for table_schema in GetTables
support actual schema
it compiles / does not fail
* chore: resolve merge conflict
* chore: make table_schema optional
* test: update e2e test for `include_schema` = true
* chore: remove info!() and update test `flightsql_schema_matches`
* chore(deps): Bump rustix from 0.36.11 to 0.37.3 (#7308 )
* chore(deps): Bump rustix from 0.36.11 to 0.37.3
Bumps [rustix](https://github.com/bytecodealliance/rustix ) from 0.36.11 to 0.37.3.
- [Release notes](https://github.com/bytecodealliance/rustix/releases )
- [Commits](https://github.com/bytecodealliance/rustix/compare/v0.36.11...v0.37.3 )
---
updated-dependencies:
- dependency-name: rustix
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* chore: make real error for existing columns
* chore: user match instead of unwrap() on column names
* chore: use datafusion::physical_plan::collect() to get record batches
* chore: use `concat_batches` to combine multiple batches into single one and fix db schema test
* chore: add doc comment for GetTables
* chore: remove pretty print
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-24 14:59:30 +00:00
3fe8bda51a
feat: Record why compact, split, or nothing plans were picked
...
And display the reason in the simulator tests to ensure decisions are
consistent.
Fixes influxdata/idpe#17306 .
2023-03-23 16:58:17 -04:00
c82674e88d
fix: Parse empty single-line comments ( #7312 )
2023-03-23 20:22:49 +00:00
99b5e0b6ec
feat: add version and uuid to startup message ( #7264 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 16:55:17 +00:00
feae0ce345
feat(router): configurable replication factor
...
This change allows the replication factor to be specified at runtime,
defaulting to "replication disabled".
2023-03-23 17:48:42 +01:00
125fef388c
feat: MVP replication support
...
This commit implements replication for the router's RpcWrite handler.
The desired number of replica copies is specified at startup time, and
each user write will be fanned-out with the specified replication factor
(replicas + 1).
A failure to write to any upstreams returns the write error, but a
failure to obtain enough ACKs (enough successful writes) after at least
1 ACK will return a "partial write" error - this differentiation is
important, as the user's write will be readable after a partial write
error has occurred.
This currently writes to upstreams serially; this is clearly an
opportunity for improvement! A follow-on PR will parallelise writes
across the desired number of replicas while maintaining the "at most one
ack'd write to one host" invariant.
Note that replication is currently hard-coded as disabled.
2023-03-23 17:48:41 +01:00
520140c322
Merge pull request #7315 from influxdata/dom/rpc-timeout-error
...
feat(router): preserve RPC error response in user response
2023-03-23 16:47:39 +00:00
53a3264d14
Merge branch 'main' into dom/rpc-timeout-error
2023-03-23 16:07:04 +00:00
1296eab093
Merge pull request #7306 from influxdata/cn/refactor
...
refactor: Organize compactor component responsibilities
2023-03-23 15:34:29 +00:00
acb491aa98
Merge branch 'main' into cn/refactor
2023-03-23 15:27:46 +00:00
2379f00929
test: set max_l0_created_at appropriately for tests of many_files ( #7304 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 15:27:28 +00:00
98ec67c564
docs: Add suggested clarification comment
2023-03-23 11:20:06 -04:00
eb232cadc8
test: set max_l0_created_at approriately for tests in single_timestamp ( #7305 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 15:19:31 +00:00
efa9732ff1
docs: Copy edit doc comments
2023-03-23 11:15:35 -04:00
e899dc70c0
feat: meaningful RPC write failure errors
...
Whenever an RPC write to an upstream ingester fails, it is retried after
an increasing delay, until the RPC_TIMEOUT is hit. Because of this, any
RPC write error would be returned as a "timeout", masking the underling
reason the write actually failed.
This commit pushes down the timeout logic, and retains the most recently
observed RPC write error, returning it to the user instead of the
timeout error.
2023-03-23 16:01:59 +01:00
e885eb9061
test(router): RPC error responses from RpcWrite
...
This PR changes the RpcWrite DmlHandler to facilitate testing using
mocked RPC client circuit breaker states, and assert the error response
for no healthy upstreams.
2023-03-23 16:01:58 +01:00
f3d96b6ed1
refactor(mock): iter for MockWriteClient returns
...
Instead of accepting a finite VecDeque of return values, allow the
MockWriteClient to take any dyn Iterator that yields the required
response type.
2023-03-23 16:01:57 +01:00
68f948c70b
refactor: rename generic type parameter
...
Rename C -> T, to be consistent with the Balancer, and open up C for
consistent usage between the two.
2023-03-23 16:01:57 +01:00
e929053109
test(fligthsql): Schema test coverage for GetTables table_schema=true ( #7313 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 14:13:42 +00:00
23ae97cd17
Merge pull request #7314 from influxdata/savage/6995-namespace-cli-should-allow-deletion
...
feat(cli): Add namespace delete command
2023-03-23 13:54:32 +00:00
d5aa360ef0
Merge branch 'main' into savage/6995-namespace-cli-should-allow-deletion
2023-03-23 13:47:37 +00:00
28bab6899a
test: set max_l0_created_at to more appropriately value for large_overlaps tests ( #7303 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-03-23 13:47:16 +00:00
8471baf567
refactor(cli): e2e test assert namespace not in list after deletion
2023-03-23 13:43:43 +00:00
7e553e73f4
Merge pull request #7288 from influxdata/dom/upstream-set-non-empty
...
refactor: non-empty RPC endpoint sets
2023-03-23 12:55:21 +00:00
744c3e05f6
Merge branch 'main' into dom/upstream-set-non-empty
2023-03-23 12:47:56 +00:00
f029a42ac1
Merge pull request #7302 from influxdata/savage/add-hot-partition-persistence-metric
...
feat(persist): Add hot partition persistence counter metric
2023-03-23 12:31:15 +00:00
Dom
dependabot[bot]
Joe-Blount
Andrew Lamb
Christopher M. Wolff
kodiakhq[bot]
Carol (Nichols || Goulding)
Chunchun Ye
Stuart Carnie
Nga Tran
Fraser Savage