9e3f7611bb
feat(ingester): buffered partition limit
...
This commit adds an optional (disabled by default) limit on the number
partitions that may be buffered for a namespace at any one time.
The exact value is configurable by setting
INFLUXDB_IOX_MAX_PARTITIONS_PER_NAMESPACE to a non-zero value, and is
disabled unless specified.
2023-09-13 17:14:43 +02:00
7f20b0faa0
chore(deps): Bump bytes from 1.4.0 to 1.5.0 ( #8692 )
...
Bumps [bytes](https://github.com/tokio-rs/bytes ) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/tokio-rs/bytes/releases )
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.4.0...v1.5.0 )
---
updated-dependencies:
- dependency-name: bytes
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-08 12:17:12 +00:00
158e2a108d
chore(deps): Bump tempfile from 3.7.1 to 3.8.0 ( #8527 )
...
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.7.1 to 3.8.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.7.1...v3.8.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-21 08:52:19 +00:00
7094189004
chore(deps): Bump tokio from 1.31.0 to 1.32.0 ( #8507 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.31.0 to 1.32.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.31.0...tokio-1.32.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:06:29 +00:00
34b8585931
chore(deps): Bump tokio from 1.30.0 to 1.31.0 ( #8482 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.30.0 to 1.31.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.30.0...tokio-1.31.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-14 06:32:34 +00:00
3675043585
chore(deps): Bump tokio from 1.29.1 to 1.30.0 ( #8464 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.29.1 to 1.30.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.29.1...tokio-1.30.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 07:50:18 +00:00
0e478af470
chore(deps): Bump tempfile from 3.7.0 to 3.7.1 ( #8429 )
...
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.7.0 to 3.7.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.7.0...v3.7.1 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-07 09:21:11 +00:00
d8badbe9ca
feat(ingester): optionally enable gossip subsystem
...
Allows the ingester to optionally enable and start the gossip subsystem
(disabled by default).
No code uses the gossip system, so no application-level messages are
exchanged, but this allows the gossip subsystem to run and exchange
control frames / perform discovery / etc.
2023-07-31 11:01:30 +02:00
32414acb00
test(bench): ingester query partition pruning
...
Adds benchmarks that exercise partition pruning during query execution
within the ingester, for varying partition counts within a table, and
varying row counts within each partition.
2023-07-24 17:26:48 +02:00
0d0f07b34e
chore(deps): Bump tempfile from 3.6.0 to 3.7.0 ( #8297 )
...
* chore(deps): Bump tempfile from 3.6.0 to 3.7.0
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.6.0 to 3.7.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.6.0...v3.7.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: Run cargo hakari tasks
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-07-21 09:06:34 +00:00
a2ca5ca17c
Merge branch 'main' into savage/hook-up-wal-reference-counter-actor
2023-07-17 10:49:45 +01:00
729851be58
test(ingester): Integration test for RPC write trace context inheritrance
2023-07-12 15:48:41 +01:00
bea9dbf7ab
test(ingester): Integration test dropping of persisted WAL segments
...
This adds an integration test that writes some data to the ingester,
waits for the WAL to be rotated and then ensures that the segment file
has been dropped.
2023-07-06 12:12:58 +01:00
70a6e60415
Merge branch 'main' into savage/use-u64-for-sequence-number
2023-07-05 12:55:44 +00:00
b15c6062a9
chore(deps): Bump tokio from 1.28.2 to 1.29.0 ( #8100 )
...
Bumps [tokio](https://github.com/tokio-rs/tokio ) from 1.28.2 to 1.29.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases )
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.28.2...tokio-1.29.0 )
---
updated-dependencies:
- dependency-name: tokio
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 13:18:08 +00:00
62cb6594c8
refactor(ingester): Use unsigned sequence number, remove its `Sqlx::Type`
...
Now that sequence numbers are internal to the ingester and the WAL,
there's no need for them to be a signed integer. As noted by
[#7260 ](https://github.com/influxdata/influxdb_iox/issues/7260 ) this was
a quirk related to the kafka-based IOx and Postgres only supported
signed integers.
2023-06-23 16:39:11 +01:00
ac26ceef91
feat: Make a place to do partition template validation
...
- Create data_types::partition_template::ValidationError
- Make creation of NamespacePartitionTemplateOverride and
TablePartitionTemplateOverride fallible
- Move SerializationWrapper into a module to make its inner field
private to force creation through one fallible constructor; this is
where the validation logic will go to be shared among all uses of
partition templates
2023-06-07 11:38:12 -04:00
7b6efae62c
chore(deps): Bump tempfile from 3.5.0 to 3.6.0
...
Bumps [tempfile](https://github.com/Stebalien/tempfile ) from 3.5.0 to 3.6.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md )
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.5.0...v3.6.0 )
---
updated-dependencies:
- dependency-name: tempfile
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-06-07 08:21:40 +00:00
9e0570f2bf
refactor: explicit submod for partition_template
...
Move the import into the submodule itself, rather than re-exporting it
at the crate level.
This will make it possible to link to the specific module/logic.
2023-05-30 15:13:20 +02:00
1ff76b7bf2
chore: use workspace dependencies for `object_store`
2023-05-26 07:03:42 -04:00
efc817c2a8
fix: Remove From impl, leaving TablePartitionTemplateOverride::new as only creation mechanism
...
This makes it clearer that you do or do not have a custom table override
(in the first argument to `new`).
2023-05-24 10:36:52 -04:00
9c0faa66f0
feat: Set a table partition template explicitly or from the namespace
...
And use the table partition template when partitioning writes to that
table.
2023-05-24 10:34:30 -04:00
604bab9508
fix: Make Table create_or_get be only create
2023-05-24 10:34:30 -04:00
afb3838437
feat: Optionally supply the namespace partition template when creating a namespace
2023-05-24 10:10:34 -04:00
928a4d163e
build: remove unused dependencies from crates
...
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).
I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!
https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html
This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
2023-05-23 14:55:43 +02:00
6344fe8c3f
chore: Add rationale for `clippy::future_not_send` ( #7822 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2023-05-18 16:58:56 +00:00
9b211df053
test(ingester): persist & persistence metrics
...
Adds a test that asserts (manually triggered) persistence generates a
file, uploads it to object storage, inserts metadata into the catalog,
and emits various persistence metrics.
2023-05-16 14:20:30 +02:00
57bedb1c2d
refactor: Extract a test helper function to create a basic namespace
2023-05-15 14:20:38 -04:00
6aa634c1b9
Merge branch 'main' into cn/move-peas
2023-05-15 13:29:42 +01:00
5fe8affb18
refactor: accept NamespaceName with Namespace create ( #7774 )
...
Co-authored-by: Dom <dom@itsallbroken.com>
2023-05-15 10:03:55 +00:00
1770d0f4d8
fix: Move ingester-querier gRPC communication to its own crate
2023-05-12 13:28:30 -04:00
58d9c40ffd
feat: If namespace or table partition templates are specified, use those
2023-05-09 14:54:57 +02:00
0849ce6f2b
fix: Rename ingester2_test_ctx to ingester_test_ctx
2023-05-08 20:23:02 -04:00
Dom Dwyer
dependabot[bot]
Fraser Savage
kodiakhq[bot]
Carol (Nichols || Goulding)
Andrew Lamb
Andrew Lamb
Kaya Gökalp