f7ff877582
Merge branch 'main' into cn/ingester-persist-tick
2023-01-13 12:31:45 +00:00
1c9fdd38d6
chore(deps): Bump io-lifetimes from 1.0.3 to 1.0.4 ( #6584 )
...
Bumps [io-lifetimes](https://github.com/sunfishcode/io-lifetimes ) from 1.0.3 to 1.0.4.
- [Release notes](https://github.com/sunfishcode/io-lifetimes/releases )
- [Commits](https://github.com/sunfishcode/io-lifetimes/compare/v1.0.3...1.0.4 )
---
updated-dependencies:
- dependency-name: io-lifetimes
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 09:21:06 +00:00
bc030150f5
refactor: improve executor panic/error handling ( #6582 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 07:10:53 +00:00
81ffb3edb5
chore: move walk and the mutable variant to the parser crate ( #6575 )
...
It is generally a useful API to be core to the InfluxQL parser crate.
2023-01-12 21:06:06 +00:00
f56123bf30
test: Allow integration tests that should_panic to pass if TEST_INTEGRATION isn't set
2023-01-12 15:31:34 -05:00
02c7ed58a2
fix: Use up the first interval tick so we wait first and then persist
2023-01-12 14:55:58 -05:00
1c7ffb95df
test: Write a should_panic test that shows ingester is persisting when I thought it wouldn't
2023-01-12 14:55:28 -05:00
97e85a24e3
Merge pull request #6570 from influxdata/cn/persist-on-demand
...
test: API to tell an ingester to persist on demand
2023-01-12 16:58:56 +00:00
0554194923
docs: Explain persist-on-demand use case and potential limitations
2023-01-12 11:52:11 -05:00
b989e0893f
test: Make persist-on-demand test a step test and check the number of parquet files
2023-01-12 11:40:46 -05:00
e1395f4f35
fix: Move PersistNow to grpc/PersistHandler
2023-01-12 11:09:33 -05:00
642bab5db3
fix: Cloning persist handle for persist_partitions no longer needed
2023-01-12 11:05:25 -05:00
27d58efa18
feat: Implement Persist Service on ingester2
2023-01-12 11:03:12 -05:00
3a2544a7eb
feat: Define a new gRPC service for ingester persist
2023-01-12 11:03:12 -05:00
e9b3efb33d
refactor: Extract a method for making requests to the ingester onto MiniCluster
2023-01-12 11:03:10 -05:00
f6dda3f4bb
test: Create shared, non-persisting ingester2 cluster
2023-01-12 10:23:00 -05:00
061188e894
Merge pull request #6580 from influxdata/dom/shutdown-sequence
...
refactor: delegate frontend shutdown to backend
2023-01-12 14:34:28 +00:00
a66dc187ac
Merge branch 'main' into dom/shutdown-sequence
2023-01-12 14:27:29 +00:00
e2da573dcf
refactor: improve thread naming ( #6579 )
...
- name exec driver thread (instead of using the default that `thread::spawn`
gives us)
- provide number to every worker thread (both for the dedicatd executor
and for the main runtime)
- shorten thread names (current naming too long for most debug tools)
2023-01-12 14:22:49 +00:00
0d111c4672
refactor: delegate frontend shutdown to backend
...
Prior to this commit, the (happy path) shutdown sequence of an IOx
process was hard coded to:
1. Stop gRPC & HTTP servers
2. Stop backend server (i.e. ingester2)
After this commit, the execution of step 1 is delegated to the handler
for step 2; the server implementation (router / ingester / querier /
etc) now chooses when to shut down the RPC & HTTP servers.
This allows the server shutdown delegate to correctly sequence the
shutdown of all components of the IOx server. This allows ingester2 to
correctly sequence the shutdown of the query RPC server w.r.t the
graceful stop & persist, ensuring queries continue to be serviced.
2023-01-12 14:59:50 +01:00
2181df75bc
Merge pull request #6576 from influxdata/dom/revert-rust-pin
...
revert: "ci: use rust 1.66.0 for CI"
2023-01-12 12:23:19 +00:00
1fa2f3595a
Merge branch 'main' into dom/revert-rust-pin
2023-01-12 12:14:39 +00:00
7e3bb25815
Merge pull request #6568 from influxdata/dom/shutdown-persist
...
feat(ingester2): persist on shutdown
2023-01-12 10:22:28 +00:00
1e5b594863
Merge branch 'main' into dom/shutdown-persist
2023-01-12 10:15:34 +00:00
8565ca5137
chore(deps): Bump zstd from 0.12.1+zstd.1.5.2 to 0.12.2+zstd.1.5.2 ( #6573 )
...
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.12.1+zstd.1.5.2 to 0.12.2+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-12 09:54:28 +00:00
cc6d8709c9
chore(deps): Bump hashbrown from 0.13.1 to 0.13.2 ( #6574 )
...
Bumps [hashbrown](https://github.com/rust-lang/hashbrown ) from 0.13.1 to 0.13.2.
- [Release notes](https://github.com/rust-lang/hashbrown/releases )
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.1...v0.13.2 )
---
updated-dependencies:
- dependency-name: hashbrown
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-12 09:47:15 +00:00
38eeffac0d
revert: "ci: use rust 1.66.0 for CI"
...
This reverts commit 71f54528d0
2023-01-12 10:36:26 +01:00
fa0893819c
fix: have warm compaction work with compactor2 ( #6571 )
...
* refactor: same function to select partition candidates
* fix: have warm compaction work with compactor2
* fix: format
* chore: cleanup
2023-01-12 02:32:39 +00:00
1f508b76fc
refactor: same function to select partition candidates ( #6569 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-12 01:14:49 +00:00
d3b2203560
fix: bug in count processed partittions ( #6572 )
2023-01-11 22:53:52 +00:00
e4249f6a19
Merge pull request #6541 from influxdata/cn/flaky-test
...
test: Record number of Parquet files before a write
2023-01-11 17:52:01 +00:00
be7c312033
fix: Wait for a particular number of Parquet files, not just any change
2023-01-11 12:11:56 -05:00
7e921e6a23
fix: Make recording num parquet files an explicit test step
...
To support a case where someone calls WriteLineProtocol twice in
a row to simulate two write requests. The test should be able to
record this state before the two write requests and not twice.
2023-01-11 11:51:56 -05:00
6677ae5c61
test: Record number of Parquet files before a write
...
Fixes #6506 .
Also has the pleasant side effect of making this code simpler and less
hacky-- it now checks the number of Parquet files for the whole
namespace, which is useful in cases where the line protocol writes to
several tables.
2023-01-11 11:51:55 -05:00
adc5c2bf06
feat: Add a gRPC API to the catalog service to get Parquet files by namespace
...
Tests that write line protocol (that may contain writes to multiple
tables) need to be able to see when new Parquet files are saved.
2023-01-11 11:41:09 -05:00
0dd4afba74
Merge branch 'main' into dom/shutdown-persist
2023-01-11 16:16:30 +00:00
be278dc689
chore(deps): Bump prost-types from 0.11.5 to 0.11.6 ( #6559 )
...
Bumps [prost-types](https://github.com/tokio-rs/prost ) from 0.11.5 to 0.11.6.
- [Release notes](https://github.com/tokio-rs/prost/releases )
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.5...v0.11.6 )
---
updated-dependencies:
- dependency-name: prost-types
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-11 16:16:04 +00:00
303c9e4398
test: fix e2e test
...
This test was relying on a graceful shutdown of the ingester to drive a
WAL replay, restoring the buffered state at startup.
Now the shutdown causes the data to be persisted and not replayed, this
didn't work.
2023-01-11 17:15:04 +01:00
2aabd5b587
Merge pull request #6567 from influxdata/dom/fix-node-discovery-shutdown
...
fix: node discovery shutdown panic
2023-01-11 16:06:40 +00:00
c01d5566b9
docs: reference WAL segement ref-counting ticket
...
Reference the WAL ref-counting ticket in the WAL rotation task.
2023-01-11 16:50:24 +01:00
bbd41228bc
feat(ingester2): persist on shutdown
...
Persist all buffered data when gracefully stopping an ingester2
instance.
This implementation accounts for both late-arriving writes, and
concurrent persist tasks - it's carefully constructed in a way that it
can discover the presence of, and wait for, outstanding persist tasks
started by other code without having to know about all the possible
places a persist task can be started (currently WAL rotation & hot
partition persistence, but later also a RPC endpoint).
There exists a small race that seems to be so incredibly unlikely to
occur, I didn't cover off (it would have a RPC write cost for little
gain). This is documented in the code comments.
2023-01-11 16:50:23 +01:00
2fa12622c8
refactor: drop PersistState visibility
...
This is now an implementation detail internal to the persist subsystem.
2023-01-11 16:42:50 +01:00
6ef68513d9
fix: gRPC balancer shutdown panic
...
The gRPC node discovery hack spawns a task that outlives the gRPC
balancer - once the balancer stops, the task should stop too (and not
panic sending on the closed channel).
2023-01-11 16:42:39 +01:00
23280d0489
refactor: return persisted partition count
...
When calling persist_partitions(), return the number of partitions that
were persisted to the caller.
2023-01-11 16:31:40 +01:00
a80e66336b
refactor: avoid unnecessary PersistHandle clone
...
Avoid having to clone a persist handle in order to pass it into
persist_partitions().
2023-01-11 16:30:58 +01:00
5148e3343d
Merge pull request #6565 from influxdata/dom/fix-ci
...
ci: use rust 1.66.0 for CI
2023-01-11 15:10:21 +01:00
5639db7c84
docs: ref-link lack of 1.66.1 issue
...
Link to the upstream issue.
2023-01-11 14:21:19 +01:00
71f54528d0
ci: use rust 1.66.0 for CI
...
Temporary fix as there's no official 1.66.1 docker image (yet?), so use
1.66 for now.
I kept the toolchain version pinned and switched only the CI version so
that we are all using the fixed cargo. I will revert this commit when
there's a 1.66.1 image.
2023-01-11 14:05:17 +01:00
a00aebfea2
chore: exclude arrow and datafusion from dependabot ( #6564 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-11 11:18:34 +00:00
93a16c78d7
chore(deps): Bump bzip2 from 0.4.3 to 0.4.4 ( #6563 )
...
Bumps [bzip2](https://github.com/alexcrichton/bzip2-rs ) from 0.4.3 to 0.4.4.
- [Release notes](https://github.com/alexcrichton/bzip2-rs/releases )
- [Commits](https://github.com/alexcrichton/bzip2-rs/commits/0.4.4 )
---
updated-dependencies:
- dependency-name: bzip2
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-11 11:11:39 +00:00
Dom
dependabot[bot]
Marco Neumann
Stuart Carnie
Carol (Nichols || Goulding)
kodiakhq[bot]
Nga Tran
Marko Mikulicic