Changes the partitioner proptest to use a timestamp generation strategy
that more accurately models the distribution of timestamps in real-world
requests.
When rendering a timestamp fails, remove the "timestamp -> generated
key" mapping from the cache.
I'm certain this is impossible to reach for multiple reasons, but it
should do the right thing anyway, in case those reasons change.
This property test generates randomised inputs and validates that all
rows with the ranges emitted by the partitioner render to the expected
timestamp-derived partition key when using a known-good implementation.
This includes generating timestamps that bypass the YYYY-MM-DD precision
reduction optimisation to increase coverage of cache miss code paths.
This fixes the root cause of influxdata/idpe#17765; the code was
performing a "is this the last value you saw" check by comparing it to
the last generated partition key which is not the same thing - a cache
hit would not generate a new key, and therefore would not return the
correct answer after.
The end result is that for a subset of writes with a problematic
sequence of timestamps would cause the wrong partition key to be
assigned. Because all users are using the default YYYY-MM-DD
partitioning scheme, the impact was relatively low, as most of the time
that partition key had the same YYYY-MM-DD representation as the last.
Partition templates should not contain more than 8 parts, which when
combined with a per-part byte limit, bounds the maximum size of a
partition key.
This commit causes the router to refuse to service a write request that
contains > 8 parts in the template - this causes a panic, as it's a
broken system invariant and should be an unreachable state. Templates
are pre-validated at creation time to contain no more than 8 parts, and
are immutable:
https://github.com/influxdata/influxdb_iox/pull/7930
This commit ensures all partition key parts are less than or equal to
200 bytes long.
If a string exceeds the 200 byte limit, it is truncated (avoiding
splitting unicode code-points or graphemes) and then a single "#"
sentinel value is appended. When reversed from the string, these column
values are indicated to be suitable for prefix-matching only - a
property that is encoded into the type system.
This commit takes a conservative approach of not splitting graphemes as
outlined in the module documentation, but this could be relaxed in the
future if needed.
Props to proptesting for this one - the prop_arbitrary_strftime_format()
randomly generated the formatting sequence "%#z" which turns out to be
an undocumented way of causing a panic in chrono:
088b69372e/src/format/mod.rs (L673)
In fact, the docs actually list is as a usable sequence!
This commit changes the partitioner to skip generating partition keys
for successive rows that would generate identical partition keys.
Often successive rows in a batch will map to the same partition key -
for example, if multiple measurements are taken at the same time, then
the strftime formatter will output the same partition key part for each
row.
This commit changes the partitioner to only generate the first key
string in such a batch of identical keys. This is cheap to pre-compute,
as we only allow tag & time columns to be partitioned, both of which are
64-bit integers (dictionary key & timestamp respectively), making it
cheaper to check equality than to allocate & generate the partition key
string and check that.
Combined with the default YYYY-MM-DD precision reduction optimisation in
a prior commit, this optimisation is particularly effective for writes
with timestamps that span a single day (the typical case).
This change doubles the rows/s throughput for a modest 1,000 line batch,
with improvements across the board. I'd expect the performance benefit
to increase as the batch size increases, and/or as more partition
template parts are added.
Allows the StftimeFormatter to perform an equality match against a
timestamp and the last rendered timestamp, potentially after applying
the precision reduction optimisation if appropriate.
- Create data_types::partition_template::ValidationError
- Make creation of NamespacePartitionTemplateOverride and
TablePartitionTemplateOverride fallible
- Move SerializationWrapper into a module to make its inner field
private to force creation through one fallible constructor; this is
where the validation logic will go to be shared among all uses of
partition templates
This commit extracts the strftime formatting logic into it's own type,
and implements a small ring-buffer based cache containing the last 5
observed timestamps (lazily initialised).
This optimisation leverages the fact that the typical write to IOx is a
batch containing many hundreds or thousands of rows - often these rows
are measurements of multiple variables at the same timestamp; for
example, a metric scrape system will periodically read a set of metrics
and assign them all the same timestamp (the "scrape timestamp").
Because of the above, batches often contain multiple rows with the same
timestamp, so we can reduce the overhead by cacheing the resulting
partition key value for any given timestamp, eliminating the need to
re-compute it for these successive identical values. We retain the last
5 observed timestamps (FIFO) to provide a degree of "look back".
Alone the above is effective for measurements all with the same exact
timestamp, often a subset of a batch. However a further optimisation is
possible: because the default partitioning scheme (YYYY-MM-DD) operates
at a granularity of days, the timestamp precision can be reduced
(discarding hours, minutes, seconds, etc) without effecting the
resulting partition key. Therefore when the default partitioning scheme
is used, this commit will normalise timestamps to match this reduced
precision before caching, causing the cache hit & string re-use rate to
rise to 100% for batches containing measurements that span < 6 days.
This brings a ~5x improvement against a modest batch size of 1,000
lines, showing improvement across all batch sizes and partitioning
schemes (default & custom). I'd expect the performance improvement to be
even greater for larger batches.
Partition keys tend to be approximately the same size each time (and in
the default case, always exactly the same size).
This simple change reduces allocations by pre-sizing the next partition
key string to match that of the previous. This should reduce the number
of allocations needed to grow the string for ~10% throughput increase.
Changes the partitioning logic to be fallible. This prevents an invalid
partition template from causing a panic, previously possible through two
known code paths:
* TagValue formatter referencing a non-tag column
* Time formatter using an invalid strftime format string
If either occurs, the write attempt is now aborted and an error returned
to the user with a HTTP 500 status code.
Additionally unexpected partitioner errors now map to a catch-all error
instead of panicking.
This test constructs a partition key from an arbitrary selection of
pre-defined parts, and uses the resulting template to partition a write
containing an arbitrary selection of pre-defined tag columns.
Once a partition key is derived, the test asserts build_column_values()
reverses it into the original set of tag (column_name, value) tuples
present in the write.
This commit changes the format of partition keys when generated with
non-default partition key templates ONLY. A prior fixture test is
unchanged by this commit, ensuring the default partition keys remain
the same.
When a custom partition key template is provided, it may specify one or
more parts, with the TagValue template causing values extracted from tag
columns to appear in the derived partition key.
This commit changes the generated partition key in the following ways:
* The delimiter of multi-part partition keys; the character used to
delimit partition key parts is changed from "/" to "|" (the pipe
character) as it is less likely to occur in user-provided input,
reducing the encoding overhead.
* The format of the extracted TagValue values (see below).
Building on the work of custom partition key overrides, where an
immutable partition template is resolved and set at table creation time,
the changes in this PR enable the derived partition key to be
unambiguously reversed into the set of tag (column_name, column_value)
tuples it was generated from for use in query pruning logic. This is
implemented by the build_column_values() method in this commit, which
requires both the template, and the derived partition key.
Prior to this commit, a partition key value extracted from a tag column
was in the form "tagname_x" where "x" is the value and "tagname" is the
name of the tag column it was extracted from. After this commit, the
partition key value is in the form "x"; the column name is removed from
the derived string to reduce the catalog storage overhead (a key driver
of COGS). In the case of a NULL tag value, the sentinel value "!" is
inserted instead of the prior "tagname_" marker. In the case of an empty
string tag value (""), the sentinel "^" value is inserted instead of the
"tagname_-" marker, ensuring the distinction between an empty value and
a not-present tag is preserved.
Additionally tag values utilise percent encoding to encode reserved
characters (part delimiter, empty sentinel character, % itself) to
eliminate deserialisation ambiguity.
Examples of how this has changed derived partition keys, for a template
of [Time(YYYY-MM-DD), TagValue(region), TagValue(bananas)]:
Write: time=1970-01-01,region=west,other=ignored
Old: "1970-01-01-region_west-bananas"
New: "1970-01-01|west|!"
Write: time=1970-01-01,other=ignored
Old: "1970-01-01-region-bananas"
New: "1970-01-01|!|!"
This test asserts the partition key of a write derived from the default
partition key template (YYYY-MM-DD).
This test ensures that the default partition keys do not change with
subsequent changes, as these values are what are used today.
Move the import into the submodule itself, rather than re-exporting it
at the crate level.
This will make it possible to link to the specific module/logic.
This commit fixes loads of crates (47!) had unused dependencies, or
mis-configured dependencies (test deps as normal deps).
I added the "unused_crate_dependencies" to all crates to help prevent
this mess from growing again!
https://doc.rust-lang.org/beta/nightly-rustc/rustc_lint_defs/builtin/static.UNUSED_CRATE_DEPENDENCIES.html
This has the minor downside of false-positives when specifying
dev-dependencies for test/bench binaries - these are files in /test or
/benches (not normal tests). This commit includes a workaround,
importing them in lib.rs (gated by a feature flag). I think the
trade-off of better dependency management is worth it!
* chore: Update datafusion + arrow/parquet/arrow-flight to 36
* refactor: update optimize for new API
* refactor: update parquet for new API
* chore: Update more dependencies
* chore: Update to use the new buffer creation APIs
* chore: Run cargo hakari tasks
* fix: bad len
* fix: update for API change
---------
Co-authored-by: CircleCI[bot] <circleci@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* chore: Upgrade to Rust 1.68
* fix: Remove unnecessary into_iter, thanks Clippy!
* fix: Use the size of the type, not a reference to the type... oops.
Thanks clippy!
* fix: Return block directly instead of creating a variable
Thanks clippy!
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This method will be used in the new ingestion pipeline to approximate how much memory a butable batch will take to convert to arrow and persist. It is meant only as a very rough estimate to trigger persistence for hot partitions.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
dependabot[bot]
Dom
Marco Neumann
Carol (Nichols || Goulding)
Andrew Lamb
kayagokalp
Carol (Nichols || Goulding)
Paul Dix
Jake Goulding