b4c1980e58
chore: remove old TODOs
...
These have been done!
2023-01-16 18:55:17 +01:00
fa62c00002
test: persist concurrent sort key catalog updates
...
Adds an integration test of the persist system, covering:
* Node A starts a persist operation
* Node B starts a persist operation for the same partition
* Node A completes, setting the catalog sort key to a new value
* Node B attempts to update the catalog, observing the new sort key
* Node B re-compacts the data, re-uploads, and drives to completion
This scenario is/was tracked in:
https://github.com/influxdata/influxdb_iox/issues/6439
2023-01-16 18:55:17 +01:00
091b428d4f
refactor(persist): decouple Context & worker logic
...
The persist::Context struct carries the data to be persisted, a
reference to the partition from which it came, and various cached fields
to avoid re-acquiring the partition read lock all the time.
Prior to this commit, the Context also had the full persist logic as
methods, invoked by the persist worker. This tightly couples the data &
logic - it's fairly clear a worker should implement the work, and
operate on the data - not commingling the two. I even knew the mess I
was making when I wrote it, but effectively copy-pasted it from
ingester1 because deadlines.
This commit decouples the persist logic from the Context.
2023-01-16 18:36:17 +01:00
1f5294c096
test: persistence system integration test
...
This test ensures the persistence system as a whole works in the happy
path.
2023-01-16 13:34:33 +01:00
5816216ba1
Merge pull request #6597 from influxdata/dependabot/cargo/rustyline-10.1.0
...
chore(deps): Bump rustyline from 10.0.0 to 10.1.0
2023-01-16 10:21:28 +00:00
586912ad73
Merge branch 'main' into dependabot/cargo/rustyline-10.1.0
2023-01-16 10:04:47 +00:00
bb6b273a93
Merge pull request #6598 from influxdata/dependabot/cargo/clap-4.1.1
...
chore(deps): Bump clap from 4.0.32 to 4.1.1
2023-01-16 10:04:17 +00:00
f7797b0358
chore(deps): Bump clap from 4.0.32 to 4.1.1
...
Bumps [clap](https://github.com/clap-rs/clap ) from 4.0.32 to 4.1.1.
- [Release notes](https://github.com/clap-rs/clap/releases )
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md )
- [Commits](https://github.com/clap-rs/clap/compare/v4.0.32...v4.1.1 )
---
updated-dependencies:
- dependency-name: clap
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-16 09:31:58 +00:00
3b2c172dc1
chore(deps): Bump nom from 7.1.2 to 7.1.3 ( #6596 )
...
Bumps [nom](https://github.com/Geal/nom ) from 7.1.2 to 7.1.3.
- [Release notes](https://github.com/Geal/nom/releases )
- [Changelog](https://github.com/rust-bakery/nom/blob/7.1.3/CHANGELOG.md )
- [Commits](https://github.com/Geal/nom/compare/7.1.2...7.1.3 )
---
updated-dependencies:
- dependency-name: nom
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-01-16 09:29:09 +00:00
1f18738203
chore: Run cargo hakari tasks
2023-01-16 02:07:11 +00:00
0a70e9f43f
chore(deps): Bump rustyline from 10.0.0 to 10.1.0
...
Bumps [rustyline](https://github.com/kkawakam/rustyline ) from 10.0.0 to 10.1.0.
- [Release notes](https://github.com/kkawakam/rustyline/releases )
- [Changelog](https://github.com/kkawakam/rustyline/blob/master/History.md )
- [Commits](https://github.com/kkawakam/rustyline/compare/v10.0.0...v10.1.0 )
---
updated-dependencies:
- dependency-name: rustyline
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-01-16 02:06:24 +00:00
3f6bb3e330
feat: Parse IANA timezones in an InfluxQL TZ clause ( #6585 )
...
* feat: Parse IANA timezone strings to chrono_tz::Tz
* feat: Visitors can customise the return error type
This avoids having to remap errors from `&'static str` to the caller's
error type, and will be used in a future PR for time range expressions.
* chore: Run cargo hakari tasks
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
2023-01-15 22:00:41 +00:00
e596f5f074
chore: metrics for compaction candidate counts ( #6593 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 19:00:43 +00:00
0edda47af2
Merge pull request #6591 from influxdata/dom/ingester-id
...
refactor: use system-wide ingester ID
2023-01-13 16:17:50 +00:00
6413362c72
refactor: use system-wide ingester ID
...
The query API exposes a unique-per-instance UUID to allow callers to
detect a crash of the ingester process - this was initialised directly
in the query RPC handler.
This commit turns the bare UUID into a type, and initialises it in the
top-level initialisation of the ingester, plumbing it down into the
query RPC handler.
This allows the UUID to be reused by other components/handlers.
2023-01-13 16:46:38 +01:00
550cea8bc5
perf: optimize not to update partitions with newly created level 2 files ( #6590 )
...
* perf: optimize not to update partitions with newly created level 2 files
* chore: cleanup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 14:46:58 +00:00
677de283ea
Merge pull request #6589 from influxdata/dom/remove-partition-double-ref
...
perf: remove double-ref Partition map
2023-01-13 14:36:28 +00:00
8dc18a9838
perf: remove double-ref Partition map
...
The ingester no longer needs to access a specific PartitionData by ID
(they are addressed either via an iterator over the BufferTree, or
shared by Arc reference).
This allows us to remove the extra map maintaining ID -> PartitionData
references, and the shared access lock protecting it.
2023-01-13 14:05:30 +01:00
81a4439cdf
Merge pull request #6583 from influxdata/cn/ingester-persist-tick
...
fix: Wait the interval and *then* persist
2023-01-13 12:39:38 +00:00
f7ff877582
Merge branch 'main' into cn/ingester-persist-tick
2023-01-13 12:31:45 +00:00
1c9fdd38d6
chore(deps): Bump io-lifetimes from 1.0.3 to 1.0.4 ( #6584 )
...
Bumps [io-lifetimes](https://github.com/sunfishcode/io-lifetimes ) from 1.0.3 to 1.0.4.
- [Release notes](https://github.com/sunfishcode/io-lifetimes/releases )
- [Commits](https://github.com/sunfishcode/io-lifetimes/compare/v1.0.3...1.0.4 )
---
updated-dependencies:
- dependency-name: io-lifetimes
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 09:21:06 +00:00
bc030150f5
refactor: improve executor panic/error handling ( #6582 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-13 07:10:53 +00:00
81ffb3edb5
chore: move walk and the mutable variant to the parser crate ( #6575 )
...
It is generally a useful API to be core to the InfluxQL parser crate.
2023-01-12 21:06:06 +00:00
f56123bf30
test: Allow integration tests that should_panic to pass if TEST_INTEGRATION isn't set
2023-01-12 15:31:34 -05:00
02c7ed58a2
fix: Use up the first interval tick so we wait first and then persist
2023-01-12 14:55:58 -05:00
1c7ffb95df
test: Write a should_panic test that shows ingester is persisting when I thought it wouldn't
2023-01-12 14:55:28 -05:00
97e85a24e3
Merge pull request #6570 from influxdata/cn/persist-on-demand
...
test: API to tell an ingester to persist on demand
2023-01-12 16:58:56 +00:00
0554194923
docs: Explain persist-on-demand use case and potential limitations
2023-01-12 11:52:11 -05:00
b989e0893f
test: Make persist-on-demand test a step test and check the number of parquet files
2023-01-12 11:40:46 -05:00
e1395f4f35
fix: Move PersistNow to grpc/PersistHandler
2023-01-12 11:09:33 -05:00
642bab5db3
fix: Cloning persist handle for persist_partitions no longer needed
2023-01-12 11:05:25 -05:00
27d58efa18
feat: Implement Persist Service on ingester2
2023-01-12 11:03:12 -05:00
3a2544a7eb
feat: Define a new gRPC service for ingester persist
2023-01-12 11:03:12 -05:00
e9b3efb33d
refactor: Extract a method for making requests to the ingester onto MiniCluster
2023-01-12 11:03:10 -05:00
f6dda3f4bb
test: Create shared, non-persisting ingester2 cluster
2023-01-12 10:23:00 -05:00
061188e894
Merge pull request #6580 from influxdata/dom/shutdown-sequence
...
refactor: delegate frontend shutdown to backend
2023-01-12 14:34:28 +00:00
a66dc187ac
Merge branch 'main' into dom/shutdown-sequence
2023-01-12 14:27:29 +00:00
e2da573dcf
refactor: improve thread naming ( #6579 )
...
- name exec driver thread (instead of using the default that `thread::spawn`
gives us)
- provide number to every worker thread (both for the dedicatd executor
and for the main runtime)
- shorten thread names (current naming too long for most debug tools)
2023-01-12 14:22:49 +00:00
0d111c4672
refactor: delegate frontend shutdown to backend
...
Prior to this commit, the (happy path) shutdown sequence of an IOx
process was hard coded to:
1. Stop gRPC & HTTP servers
2. Stop backend server (i.e. ingester2)
After this commit, the execution of step 1 is delegated to the handler
for step 2; the server implementation (router / ingester / querier /
etc) now chooses when to shut down the RPC & HTTP servers.
This allows the server shutdown delegate to correctly sequence the
shutdown of all components of the IOx server. This allows ingester2 to
correctly sequence the shutdown of the query RPC server w.r.t the
graceful stop & persist, ensuring queries continue to be serviced.
2023-01-12 14:59:50 +01:00
2181df75bc
Merge pull request #6576 from influxdata/dom/revert-rust-pin
...
revert: "ci: use rust 1.66.0 for CI"
2023-01-12 12:23:19 +00:00
1fa2f3595a
Merge branch 'main' into dom/revert-rust-pin
2023-01-12 12:14:39 +00:00
7e3bb25815
Merge pull request #6568 from influxdata/dom/shutdown-persist
...
feat(ingester2): persist on shutdown
2023-01-12 10:22:28 +00:00
1e5b594863
Merge branch 'main' into dom/shutdown-persist
2023-01-12 10:15:34 +00:00
8565ca5137
chore(deps): Bump zstd from 0.12.1+zstd.1.5.2 to 0.12.2+zstd.1.5.2 ( #6573 )
...
Bumps [zstd](https://github.com/gyscos/zstd-rs ) from 0.12.1+zstd.1.5.2 to 0.12.2+zstd.1.5.2.
- [Release notes](https://github.com/gyscos/zstd-rs/releases )
- [Commits](https://github.com/gyscos/zstd-rs/commits )
---
updated-dependencies:
- dependency-name: zstd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-12 09:54:28 +00:00
cc6d8709c9
chore(deps): Bump hashbrown from 0.13.1 to 0.13.2 ( #6574 )
...
Bumps [hashbrown](https://github.com/rust-lang/hashbrown ) from 0.13.1 to 0.13.2.
- [Release notes](https://github.com/rust-lang/hashbrown/releases )
- [Changelog](https://github.com/rust-lang/hashbrown/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/hashbrown/compare/v0.13.1...v0.13.2 )
---
updated-dependencies:
- dependency-name: hashbrown
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dom <dom@itsallbroken.com>
2023-01-12 09:47:15 +00:00
38eeffac0d
revert: "ci: use rust 1.66.0 for CI"
...
This reverts commit 71f54528d0
2023-01-12 10:36:26 +01:00
fa0893819c
fix: have warm compaction work with compactor2 ( #6571 )
...
* refactor: same function to select partition candidates
* fix: have warm compaction work with compactor2
* fix: format
* chore: cleanup
2023-01-12 02:32:39 +00:00
1f508b76fc
refactor: same function to select partition candidates ( #6569 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-01-12 01:14:49 +00:00
d3b2203560
fix: bug in count processed partittions ( #6572 )
2023-01-11 22:53:52 +00:00
e4249f6a19
Merge pull request #6541 from influxdata/cn/flaky-test
...
test: Record number of Parquet files before a write
2023-01-11 17:52:01 +00:00
Dom Dwyer
dependabot[bot]
CircleCI[bot]
Stuart Carnie
Nga Tran
kodiakhq[bot]
Marco Neumann
Carol (Nichols || Goulding)