Record the number of tables in each write - this will let us observe the
total number of tables a router instance has observed, which when
combined with the existing metrics helps us understand the shape
(distribution of tables/lines/fields) of the workload hitting the
routers.
Uses the new ColumnRepo::create_or_get_many() catalog method to perform
a bulk upsert of (potentially) new columns to the catalog during schema
validation.
A quick change to perform the ColumnRepo::create_or_get() calls in
parallel (up to a maximum of 3 in-flight at any one time) in order to
mitigate the latency of the call and reduce the overall schema
validation call duration.
The in-flight limit is enforced to avoid starving the DB connection pool
of connections.
Adds two new metrics:
* namespace_cache_table_count: total number of tables in cache
* namespace_cache_column_count: total number of columns in cache
The metric decorator keeps a running total of each of the table and
column counts as namespaces are inserted into the cache, and adjusts the
value accordingly when an existing namespace is overwrote.
Changes the configuration of the router request pipeline to move schema
validation before partitioning.
This reduces the concurrency of callsm into the schema validator when a
single write is split into one or more partitions, reducing contention
and cash thrashing. It also ensures we don't bother partitioning the
writes if the request will fail.
Configures the instrumentation decorator to emit a trace span covering
the duration of the decorated handler's execution, recording the
success/error result and and error message, if any.
Adds an integration test covering the router's HTTP handler stack.
Given a well-formed HTTP write, the test asserts:
* Write passes through the stack without error
* Response code sent to client
* Write buffer message is enqueued
* Catalog namespace record is created
* Metric handler is invoked and the hit is recorded
The router is composed of several DML handlers called in sequence in
order to construct the full request handling pipeline. Prior to this
commit, each handler nested the next handler it calls internally,
producing a nested call chain that resulted metrics (added in #3764)
recording cumulative latency like this:
┌ ─
│ ┌───────────────┐
│ NS Creation │
│ └───────────────┘
│ ┌───────────────┐
│ │ │ Partitioner │
│ └───────────────┘
│ │ │
│ │
Cumulative │ │ │ ┌───────────────┐
Timings 1.5s 1s │ etc... │
│ │ │ └───────────────┘
│ │
│ │ │
│ ┌───────────────┐
│ │ │ Partitioner │
│ └───────────────┘
│ ┌───────────────┐
│ NS Creation │
│ └───────────────┘
└ ─
This meant it was hard to determine the latency of a single handler
without knowing (and subtracting the latency of) all the child handlers
it calls.
This commit replaces the intrusive nested handler call chain with an
external Chain combinator type to compose together individual handlers,
resulting in correct per-handler timings and simpler code/tests:
┌───────────────┐
│ NS Creation │
└───────────────┘
│
.5s ┌───────────────┐
└───────▶│ Partitioner │
└───────────────┘
│
1s ┌───────────────┐
└───▶│ etc... │
└───────────────┘
Changes the NamespaceAutocreation handler to be generic over any
WriteInput.
This allows the NamespaceAutocreation layer to be placed anywhere in the
handler stack, without needing a prior transformation or specific write
type.
Implements a write partitioning DML handler that splits per-table
MutableBatch instances into per-partition, per-table MutableBatch and
concurrently calls the inner DML handler with each.
Allow the MockDmlHandler to capture any input type given to the write()
method. This lets us reuse the mock across all handler implementations,
regardless of their expected write input type.
Allow a DML handler to specify the write input type on which it
operates.
This allows us to construct a write handler pipeline that transforms the
request as it passes through the various handlers. We'll use this to
implement a handler that annotates a normal set of table writes with the
partition key, modifying downstream handlers to expect this annotated
input.
* feat: allow catalog access w/o a transaction
Now the caller has the full control if they want to use a transaction or
not.
* fix: remove non-transaction-safe `create_many`
* fix: remove unnecessary transactions
* refactor: catalog Unit of Work (= transaction)
Setup an inteface to handle Units of Work within our catalog. Previously
both the Postgres and the in-mem backend used "mini-transactions on
demand". Now the caller has a clear way to establish boundaries and
gets read and write isolation. A single `Arc<dyn Catalog>` can create as
many `Box<dyn UnitOfWork>` as you like, but note that depending on the
backend you may not scale infinitely (postgres will likely impose
certain limits and the in-mem backend limits concurrency to 1 to keep
things simple).
* docs: improve wording
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* refactor: rename Unit of Work to Transaction
* test: improve `test_txn_isolation`
* feat: clearify transaction drop semantics
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds a simple wrapper type that maps the namespace keyspace over a set
of N namespace schema caches, thereby reducing cache lock contention by
a factor of N (in a perfect world).
This will help smooth out latency of workloads that include new
namespace requests or incremental schema additions. It should also
significantly help latency during initial cache warming of a freshly
booted router.
This adds the scaffolding for the ingester server to consume data from Kafka. This ingests data in an in memory structure while creating records in the catalog for any partitions that don't yet exist.
I've removed catalog_update.rs in ingester for now. That was mostly a placeholder and will be going in a combination of handler.rs and data.rs on my next PR which will have some primitive lifecycle wired up.
There's one ugly bit here where the DML write is cloned because it's getting borrowed to output spans and metrics. I'll need to follow up with a refactor to make it so that the DML write's tables can be consumed without it gumming up the metrics stuff.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds an in-memory cache of table schemas to the SchemaValidator DML
handler.
The cache pulls from the global catalog when observing a column for the
first time, and pushes the column type to set it for subsequent requests
if it does not exist (this pull & push is done by atomically by the
catalog in an "upsert" call).
The in-memory cache is sharded by namespace, with each shard guarded by
an individual lock to minimise contention between readers (the expected
average case) and writers (only when adding new columns/tables).
Relies on the catalog to serialise new column creation and validate
parallel creation requests.
Implements a write schema validation DML handler, denying requests that
conflict with the schema within the global catalog. Additive schema
changes are accepted, incrementally updating the global catalog schema.
Deletes are passed through unchanged and unvalidated.
Allows the DmlHandler to return different types for each method.
This enables a DmlHandler implementation decorating an inner handler to
return the inner handler's error directly, avoiding any "wrapper"
errors.
Dom Dwyer
Carol (Nichols || Goulding)
Marco Neumann
Luke Bond
dependabot[bot]
kodiakhq[bot]
Paul Dix
Luke Bond
Andrew Lamb