* test: regression test for #8378
* fix: avoid recursive locking during LRU shutdown
Fixes the following construct during shutdown:
1. `clean_up_loop` holds `members` lock
2. calls `member.remove_keys`
3. `CallbackHandle::execute_requests` requests upgrades weak ref and gets lock
4. other thread drops last external reference to pool member, the
upgraded weak ref from (3) is now the last strong ref
5. `CallbackHandle::execute_requests` finishes, drops pool member
6. dropping that pool member calls `ResourcePool::unregister_member`
which is the same lock as we got in (1) => deadlock
We now just avoid modifying `members` during shutdown and just hold a
weak ref there. As a side effect, the `last_used` addressable heap moves
around a bit an is no longer `Arc`ed (see updated diagram).
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
chore(doc): fix typos and format
chore(doc): fix format
chore(doc): rename file
chore(doc): add new file to doc README
chore(doc): format
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Now that PartitionId is only sent to the querier if there's no hash id
in the catalog, the query adaptor doesn't need to hold onto the catalog
partition ID if we know we're in that case (which the bloom filter will
tell us soon).
The catalog ID shouldn't be used anywhere, as the two fields get turned
into a TransitionPartitionId on the querier side.
This will enable us to not query the catalog if we're sure this
partition has a hash ID in the catalog.
Also rename PartitionInfo's transition_partition_id to be partition_id
so that it's consistent with the QueryChunk method. We might want to
rename the partition_id field to catalog_partition_id, but for now I
think the types will make compactor usage clear enough.
This gets compactor to compile and pass its tests.
Thus using the PartitionHashId if one is available. This does not
compile yet because of all the uses of QueryChunk, but iox_query
compiles and passes its tests.
I'm going to introduce a new ARBITRARY_PARTITION_ID that will be a
TransitionPartitionId, but there will still be some tests that need a
PartitionId, so rename to disambiguate.
This commit adds the NamespaceSchemaGossip type, a decorator of
[`NamespaceCache`] implementations utilising peer gossiping to provide
best-effort convergence of the local cache state.
This decorator will sit in the NamespaceCache stack, allowing it to
receive incoming schema gossip messages, and update the local cache
through the regular NamespaceCache abstraction methods.
This currently implements the message handlers only - no messages are
sent yet!
* refactor: use `Box<[...]>` instead of `Vec<...>`
We are not planning to modify the vector, so storing a capacity and a
length is somewhat pointless.
* feat: add printout test for PG migrations
* refactor: use dedicated checksum type
* feat: checksum string roundtrips
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
This benchmark covers two axis of performance for calls to the
namespace cache's `put_schema()` stack. These are the cost of adding
varying numbers of new columns to an existing table in the namespace, as
well as adding new tables with their own set of columns to an existing
namespace.
Not all use cases will involve sending messages (some will only want to
subscribe to messages) which might result in someone dropping the handle
they're not expecting to use.
Forgot to update the docs in #8373.
This will be updated again after I've finished #7897.
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Adds the supporting types required to integrate the generic gossip crate
into a schema-specific broadcast primitive.
This commit implements the two "halves":
* GossipMessageDispatcher: async processing of incoming gossip msgs
* Handle: the send-side handle for async sending of gossip msgs
These types are responsible for converting into/from the serialised
bytes sent over the gossip primitive into application-level / protobuf
types.
* chore(cli): add `--partition-template` to namespace create
* chore: fix typo in doc for `PartitionTemplateConfig`
chore: add max limit 8 for partition template in doc
* chore: add e2e tests
* chore: fmt
* chore: add more e2e tests for namespace create with partition template
* chore: show doc comments in cli help interface
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* feat: add sort_key_ids as array of bigints into catalog partition
* chore: add comments
* chore: remove comments to avoid changing them in the future due to checksum requirement
---------
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Andrew Lamb
Marco Neumann
Chunchun Ye
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
kodiakhq[bot]
Fraser Savage
Dom Dwyer
dependabot[bot]
Nga Tran