f419e2c378
feat: warm compaction ( #6192 )
...
* feat: warm compaction
chore: add missing warm compaction config
chore: tests for warm compaction
chore: modify count usage in warm compaction sql
chore: catalog test for warm compaction; sql fixes
feat: settable target level for compact w/ budget
chore: tests for warm compaction
chore: clarifying comments in warm compaction test
chore: fixed erroneous comment in catalog test
chore: improve warm compactor test by checking file exists
chore: tests for warm compaction
chore: warm compactor test tidy-ups
* chore: improve test for warm compaction
* chore: fix erroneous comment in warm compaction code
2022-12-16 15:59:45 +00:00
2a1e540ee3
fix: Rename INFLUXDB_IOX_MODE to INFLUXDB_IOX_RPC_MODE
2022-12-15 14:13:01 -05:00
7d216ba1fd
feat: Error if you run the wrong command with the wrong env var set
...
Connects to #6402 .
2022-12-15 14:06:59 -05:00
aec98015d7
fix: Remove the rpc_write feature flag and use INFLUXDB_IOX_MODE env var instead
...
And standardize on ingester2 and router2 for consistency.
Connects to #6402 .
2022-12-15 14:06:59 -05:00
8e1bf9cdf7
feat: Increase default all-in-one compactor memory to 300MB and expose it as a setting ( #6405 )
2022-12-14 20:35:48 +00:00
66c610f7b1
Merge branch 'main' into cn/ingester-persisted-file-count
2022-12-14 14:58:31 +00:00
47cd6821e1
feat: Document IOx Flight API and add convenience methods ( #6392 )
...
* feat: Document IOx Flight API and add convenience methods
* fix: InfluxQL handling
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-13 17:32:37 +00:00
fdbf9e112e
fix: Actually switch into rpc_write mode in querier
...
Only when the feature flag is set *and* --ingester-addresses is set. I
had documented that intention, but didn't actually implement it
correctly.
2022-12-12 16:37:11 -05:00
1c7f322a4e
feat: Keep track of and report number of Parquet files persisted
...
Per partition and starting over each time the ingester restarts.
Fixes #6334 .
2022-12-12 11:45:00 -05:00
727efcbdee
Merge branch 'main' into cn/ingester2-uuid
2022-12-12 16:21:15 +00:00
e5322b24b9
feat: Add --token CLI argument, improve update docs about writing ( #6356 )
...
* feat: Add --token CLI argument, improve update docs about writing
* fix: support environment tokens too
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-12 13:43:15 +00:00
c3a7575d46
feat: Enable rpc_write on the inner command if it's enabled for tests
...
And only run rpc_write specific tests if the feature is enabled when
running the tests.
2022-12-08 17:45:30 -05:00
5141cba1db
fix: Only switch into querier RPC write path if ingester addresses specified
...
This enables testing of the querier using the old path with the
rpc_write feature turned on.
2022-12-08 17:40:04 -05:00
2fd2d05ef6
feat: Identify each run of an ingester with a Uuid
...
And send that UUID in the Flight response for queries to that ingester
run.
Fixes #6333 .
2022-12-08 17:22:52 -05:00
619a2d0856
fix: Remove conflicting arguments from the RouterRpcWriteConfig ( #6355 )
...
These were added in
https://github.com/influxdata/influxdb_iox/pull/6346 .
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 20:21:37 +00:00
4ded68de62
test: "not found" end2end tests for querier ( #6352 )
...
I couldn't find any end2end tests for these cases and I was kinda
worried that our error codes were wrong. Turns out they are correct, but
let's have some nice tests for this behavior.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-08 18:17:53 +00:00
6f7cb5ccf0
Merge branch 'main' into cn/ingester2-querier
2022-12-08 14:00:49 +00:00
e13e668d26
refactor: Share more code in the querier in the RPC write path mode
2022-12-07 13:54:08 -05:00
551bb0ef6a
feat: allow enabling/disabling ns autocreation in router ( #6346 )
...
* feat: allow enabling/disabling ns autocreation in router
* fix: missed an import for something behind router2 compile flag
2022-12-07 16:12:00 +00:00
9166ace796
feat: Make a mode for the querier to use ingester2 instead, behind the rpc_write feature flag
2022-12-07 09:56:50 -05:00
1d38d400f0
chore(deps): Bump object_store from 0.5.1 to 0.5.2 ( #6339 )
...
* chore(deps): Bump object_store from 0.5.1 to 0.5.2
Bumps [object_store](https://github.com/apache/arrow-rs ) from 0.5.1 to 0.5.2.
- [Release notes](https://github.com/apache/arrow-rs/releases )
- [Changelog](https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md )
- [Commits](https://github.com/apache/arrow-rs/compare/object_store_0.5.1...object_store_0.5.2 )
---
updated-dependencies:
- dependency-name: object_store
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-06 07:53:54 +00:00
f62b270852
fix: gRPC errors regarding group cols ( #6314 )
...
* fix: gRPC errors regarding group cols
- missing group col prev. produced an "internal error" but should be
"invalid argument"
- duplicate group cols produced a panic but should also be "invalid
argument"
* docs: clarify
2022-12-06 07:36:32 +00:00
cd6a8a1a82
refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics ( #6313 )
...
* refactor: DF-driven on-demand mem limit instead of ahead-of-time heuristics
Closes #6310 .
* refactor: rename and tune default exec mem limits
* fix: ingester2 bits after rebase
2022-12-05 12:38:28 +00:00
fc9431f6ae
chore(deps): Bump assert_cmd from 2.0.6 to 2.0.7 ( #6330 )
...
Bumps [assert_cmd](https://github.com/assert-rs/assert_cmd ) from 2.0.6 to 2.0.7.
- [Release notes](https://github.com/assert-rs/assert_cmd/releases )
- [Changelog](https://github.com/assert-rs/assert_cmd/blob/master/CHANGELOG.md )
- [Commits](https://github.com/assert-rs/assert_cmd/compare/v2.0.6...v2.0.7 )
---
updated-dependencies:
- dependency-name: assert_cmd
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-12-05 09:05:56 +00:00
66aab55534
feat(ingester2): run persistence task
...
Configures the initialisation of an ingester2 instance to spawn a
persistence task (currently unused) and plumbs in various configuration
parameters.
2022-12-02 17:18:39 +01:00
9e3d0fcefb
Merge branch 'main' into cn/ingester2
2022-12-02 13:39:55 +00:00
77cbc880f6
feat: Add cap limit on number of partitions to be compacted in parallel ( #6305 )
...
* feat: Add cap limit on number of partitions to be comapcted in parallel
* chore: cleanup
* chore: clearer comments
2022-12-01 21:23:44 +00:00
b9e424582f
refactor: Extract a clap block for the ingester2 RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
fef3bc02cd
refactor: Extract a clap block for the router RPC write path
...
To be able to share it with the coming all-in-one2 command
2022-12-01 11:39:30 -05:00
0133479122
feat: Create an ingester2 server
2022-12-01 11:39:29 -05:00
99698d0f46
feat: Create an ioxd_ingester2 crate
2022-12-01 11:37:21 -05:00
fc694e7273
feat: Add an ingester2 command behind the rpc_write feature flag
2022-12-01 11:37:21 -05:00
c008219692
feat: Add a feature flag to switch to the router RPC write path ( #6247 )
...
* feat: Add a feature flag to switch to the router RPC write path
Fixes #6242 .
* refactor: Remove a weird arc clone/rename that's not needed
I'm sure this was needed at some point, but it doesn't make much sense.
I wasn't going to change this, but I'm now trying to minimize the
differences between this function and the write path init function, so
make this one better too.
* fix: Add the namespace autocreation to the RPC write path too
The topic/query pool don't really apply to this case, but use them
anyway to be able to use the existing catalog methods.
Also add a bunch of comments pointing out where the RPC write path
initializer and the old router's initializer are the same and where
they're different, so that perhaps it'll be easier to keep them in sync
while they both exist.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-12-01 11:05:39 +00:00
d07658282c
feat: add router config parameter for retention ( #6278 )
...
* chore: remove unused/moved ns_autocreation dml handler
* feat(router): expose new ns retention as config
* fix: forgot to set default value for router retention arg
* chore: make new namespace retention param an option
2022-11-30 13:14:39 +00:00
2306c383f3
feat: Introduce InfluxQL to Flight ( #6166 )
...
* feat: Introduce InfluxQL to Flight
All InfluxQL queries will fail with an error
* chore: Temper protobuf lint
* chore: Finalize flight.proto changes; fix tests
* chore: Add tests for InfluxQL planner
* chore: Update docs
* chore: Update docs
* chore: Rename back to original
* chore: Use .into() rather than cast
* chore: Use function rather than field
* chore: Improved InfluxQL planner name
* chore: Restore `impl Into<String>` argument
* chore: Add a comment that Go clients are unable to execute InfluxQL
* chore: Add a test for the `--lang` argument and InfluxQL
2022-11-23 00:33:49 +00:00
caa595a6fc
chore(deps): Bump serde_json from 1.0.88 to 1.0.89 ( #6203 )
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.88 to 1.0.89.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.88...v1.0.89 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 09:28:31 +00:00
04c00bbb62
chore(deps): Bump bytes from 1.2.1 to 1.3.0 ( #6199 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.2.1 to 1.3.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/commits )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-22 08:23:24 +00:00
67af105bde
feat: add `process_start_time_seconds` metric ( #6187 )
...
This is helpful to reason about IOx versions and changes/alerts during
deployments.
2022-11-21 12:27:04 +00:00
a9db7581cd
chore(deps): Bump tokio from 1.21.2 to 1.22.0 ( #6183 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.21.2 to 1.22.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.22.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-21 10:21:24 +00:00
52c50c16e1
chore(deps): Bump serde_json from 1.0.87 to 1.0.88
...
Bumps [serde_json](https://github.com/serde-rs/json ) from 1.0.87 to 1.0.88.
- [Release notes](https://github.com/serde-rs/json/releases )
- [Commits](https://github.com/serde-rs/json/compare/v1.0.87...v1.0.88 )
---
updated-dependencies:
- dependency-name: serde_json
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-11-21 01:52:18 +00:00
7c813c170a
feat: reintroduce compactor first file in partition exception ( #6176 )
...
* feat: compactor ignores max file count for first file
chore: typo in comment in compactor
* feat: restore special first file in partition compaction logic; add limit
* fix: calculation in compaction max file count
chore: clippy
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 15:58:59 +00:00
49a9565240
feat: gRPC that creates namespace ( #6103 )
...
* feat: create namespace API call in router
Co-authored-by: Nga Tran <nga-tran@live.com>
* chore: treat retention as ns except in CLI
* fix: overflow in nanosecond calc
* fix: retention test after changing it from hours to ns
* chore: comment clarification in cli; better response type for error in ns API
* fix: correct some rebase mistakes
* chore: merge namespace create & create_with_retention; renamed ns create test helper fn & const
* fix: ns autocreation test was wrong after rebase
* fix: mem catalog has default 1hr retention, accidently removed in rebase
* chore: remove mem catalogs default 1hr retention; make it settable in sets & router
Co-authored-by: Luke Bond <luke.n.bond@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-18 13:02:12 +00:00
6f7b1e2e26
feat: reject writes that are outside the retention period ( #6148 )
...
* feat: reject writes that are outside the retention period
* feat: add retention validator into handler stack
* chore: Apply suggestions from code review
Co-authored-by: Dom <dom@itsallbroken.com>
* refactor: address review comments
* test: unit tests fot retention validation
* chore: address review comments
* test: more unit tests and integration tests
* refactor: make time inside retention period for emphemeral_mode test
* fix: 2 hours
Co-authored-by: Dom <dom@itsallbroken.com>
2022-11-17 20:55:58 +00:00
15191d11a8
feat: improve error message on `storage` `--format` parse errors ( #6155 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-17 14:01:26 +00:00
9365d933f1
chore: router namespace api ( #6151 )
...
* chore: move ns api from querier to router
* chore: add explanatory comment in querier about moved namespace API
* fix: add namespace service to router
* fix: querier returns unimplemented error for ns retention, not panic
* chore: reuse namespace -> proto in router ns api
* chore: grpc namespace - consume ns to avoid clone
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-16 15:25:49 +00:00
62851afc27
feat: add querier->ingester circuit breaker ( #6147 )
...
* feat: add log ingester memory pressure persist
* feat: add querier->ingester circuit breaker
Closes #4608 .
* docs: explain high-level circuit breaker
* docs: improve
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
* test: add additional test assertion
* refactor: upgrade info to warning log
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2022-11-16 10:50:33 +00:00
a969754819
chore(deps): Bump chrono from 0.4.22 to 0.4.23 ( #6129 )
...
* chore(deps): Bump chrono from 0.4.22 to 0.4.23
Bumps [chrono](https://github.com/chronotope/chrono ) from 0.4.22 to 0.4.23.
- [Release notes](https://github.com/chronotope/chrono/releases )
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md )
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.22...v0.4.23 )
---
updated-dependencies:
- dependency-name: chrono
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* refactor: chrono future compat
Integer->timstamp conversions should not silently panic.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marco Neumann <marco@crepererum.net>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2022-11-14 13:34:09 +00:00
05d7d1495e
Merge branch 'main' into dependabot/cargo/hashbrown-0.13.1
2022-11-11 21:26:40 +00:00
042bf545fe
fix: Rename database types/vars to namespace in the storage command code
2022-11-11 16:14:12 -05:00
bdff4e8848
fix: Consistently use 'namespace' instead of 'database' in comments and other internal text
2022-11-11 15:46:04 -05:00
Luke Bond
Carol (Nichols || Goulding)
Andrew Lamb
kodiakhq[bot]
Carol (Nichols || Goulding)
Marco Neumann
dependabot[bot]
Dom Dwyer
Nga Tran
Stuart Carnie