To support a case where someone calls WriteLineProtocol twice in
a row to simulate two write requests. The test should be able to
record this state before the two write requests and not twice.
Fixes#6506.
Also has the pleasant side effect of making this code simpler and less
hacky-- it now checks the number of Parquet files for the whole
namespace, which is useful in cases where the line protocol writes to
several tables.
This test was relying on a graceful shutdown of the ingester to drive a
WAL replay, restoring the buffered state at startup.
Now the shutdown causes the data to be persisted and not replayed, this
didn't work.
Persist all buffered data when gracefully stopping an ingester2
instance.
This implementation accounts for both late-arriving writes, and
concurrent persist tasks - it's carefully constructed in a way that it
can discover the presence of, and wait for, outstanding persist tasks
started by other code without having to know about all the possible
places a persist task can be started (currently WAL rotation & hot
partition persistence, but later also a RPC endpoint).
There exists a small race that seems to be so incredibly unlikely to
occur, I didn't cover off (it would have a RPC write cost for little
gain). This is documented in the code comments.
The gRPC node discovery hack spawns a task that outlives the gRPC
balancer - once the balancer stops, the task should stop too (and not
panic sending on the closed channel).
Temporary fix as there's no official 1.66.1 docker image (yet?), so use
1.66 for now.
I kept the toolchain version pinned and switched only the CI version so
that we are all using the fixed cargo. I will revert this commit when
there's a 1.66.1 image.
* feat: InfluxQL learns how to plan some queries
Also added a means to test the planner and execution
* chore: Update module docs
* chore: Document the planner functions
* chore: Update end_to_end_cases crate
* chore: Clarify why `SLIMIT` and `SOFFSET` return `NotImplemented`
* chore: Address lint issues
* chore: Fix rustdoc link issue
* chore: Remove InfluxQL tests from query_tests crate
Will follow conventions established by @carols10cents when
new query_tests crate is merged.
* chore: `now` field
`now` is a DataFusion built-in scalar function
* chore: remove unused code
* chore: Add additional arithmetic expression tests
* chore: Establish pattern for identifying and tracking InfluxQL issues
* chore: Add tests for case sensitivity issues
* chore: group tests into modules and functions
This avoids mass rewriting of insta snapshots as new
tests are added to each function. When tests are added in the middle,
existing snapshots are renamed (-N+1, -N+2, etc) resulting in
having to review numerous additional snapshots.
* feat: cold
* chore: debug info
* feat: only compact qualified cold partition candidates
* fix: catalog test
* chore: cleanup
* chore: add new config flag for cold partition candidates
* chore: implement display for CompactionType and add tests for max num partitions
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Prior to this commit, when initialising the persist system it would
return a PersistState instance, used to communicate the saturation
status of the persistence system. The RPC write path used this
information to accept or deny write requests accordingly.
This was unfortunate in that it tightly coupled the ingest handler to
the persist system - in order to initialise the RPC handler, you had to
provide a PersistState; this required us to initialise a persist system
when testing only the RPC handler (which had nothing to do with
persisting). This smells!
This commit inverts the dependency, and decouples the subsystems via a
shared type (IngestState). Instead of the persist system telling ingest
to stop, the ingest system provides a means to be told to stop - this
subtle difference decouples the ingest handler from all components that
need to block ingest. This allows a fast O(1) error state read for N
components and prevents us from having to start N components to test a
RPC handler.
Additionally this commit introduces an unused ingest error state
(GracefulStop) as part of figuring out the API (to be used shortly).
* refactor: invalidate querier cache if ingester is gone
For #6549 but I think even w/o the plan illustrated there, this is the
right thing to do.
Also changes the cache system to use flats sorted vectors instead of costly hash
maps.
* refactor: simplify code
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Dom
dependabot[bot]
Nga Tran
kodiakhq[bot]
Carol (Nichols || Goulding)
Marko Mikulicic
Marco Neumann
Stuart Carnie