Connects to #4399.
Only file-based write buffer is supported. If `--data-dir` is specified,
store it there, otherwise store it in a temp directory to be ephemeral
Connects to #4399.
Manually flatten the arguments to set different defaults, not allow
setting the partition min/max, but still allow customization of the
other arguments.
Prior to this commit, adding any metric to the catalog (and only the
catalog) would cause the end_to_end_ng_cases::metrics::test_metrics test
to fail due to asserting an exact number of metrics observed.
This commit changes the check condition to a more permisive >= rather
than ==.
Fetch all NamespaceSchema from the catalog and pre-cache them during
router initialisation.
This pre-warming happens in 3 queries, and should significantly reduce
the load on the Postgres instance backing the catalog during
deployments/after router crashes/etc.
Pre-warming occurs during init, before the HTTP endpoints are bound, and
therefore before the /health HTTP endpoint can return OK. This ensures
the router pre-warms the cache before it begins receiving traffic.
- only convert dictionary types that we really want to convert (instead
of blindly converting all types)
- handle missing / NULL columns
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: use stored sort key to deduplicate data
* refactor: verify if one is a super sort key of the other
* test: unit tests for scan and deduplication plans
* fix: typo
* refactor: refactor and add comments
* feat: cache partition sort key to read during planning as needed
* test: tests for query plans with different overlap groups
* chore: cleanup
* chore: resolve merge conflicts
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor: improve `IngesterData` public interface
* feat: impl `Debug` for `Test{Namespace,Sequencer}`
* refactor: trait interface for `LifecyleHandle`
This is required to mock the lifecycle for query tests.
* refactor: trait for partitioner
Add the "http_request_limit_rejected" metric that is incremented once
each time a request is dropped due to the simultaneous request service
protection limit being exceeded.
This metric will allow us to effectively alert on router saturation, and
the increase rate/second will help inform us in how much capacity needs
adding.
This commit adds a service protection limit, allowing the router to
restrict the number of simultaneous HTTP requests to a hard-coded limit
of 200.
By dropping additional requests once the router is serving the
configured maximum, the router is prevented from accepting more and more
requests that stack up waiting on the catalog / postgres connection
pool. Currently buffering all these stalled connections consumes so much
memory the router is OOM killed. After this commit, a portion of
requests will be satisfied and the rest will be dropped early with a
HTTP 503 Service Unavailable response.
* refactor: document and improve `MockIngesterConnection`
* refactor: split `OldOneMeasurementFourChunksWithDuplicates` for `EXPLAIN` queries
* fix: mark "IngsterPartition" chunks as unsorted
* fix: "group by" queries may require sorted comparison
* refactor: re-export a few more types from querier
* fix: ensure that test parquet files are de-duped
* test: chunks in ingester stage
* docs: explain test code
* refactor: grouping overlaps is now use the same overlap function in both compactor and deduplication
* chore: commit missing file
* chore: address review comments
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* refactor: Expose data generation tool for wider use
* feat: Add a step for retrieving the server metrics
* refactor: Copy the OG end-to-end metrics test to NG
* feat: Rewrite the NG end-to-end metrics test
This is still broken because the the row timestamp metrics don't exist
in NG.
* fix: Test metrics relevant to NG
* refactor: Move the data generator to the test helper crate
* refactor: Extract a ReadFilter request builder into the test helper crate
* refactor: Make test helper request builder able to build other gRPC requests
Co-authored-by: Carol (Nichols || Goulding) <carol.nichols@gmail.com>
Carol (Nichols || Goulding)
Andrew Lamb
kodiakhq[bot]
Dom Dwyer
dependabot[bot]
Marco Neumann
Carol (Nichols || Goulding)
Nga Tran
二手掉包工程师
Jake Goulding