The jemalloctor rust global allocator only intercepts allocations coming from rust.
We used to enabled the `unprefixed_malloc_on_supported_platforms` on the jemallocator crate
which effectivelly installs jemalloc as a replacement for system malloc so that both rust and C libraries
would use jemalloc. This not only ensures that all libraries use jemalloc for the sake of using
a better allocator, but also allows us to use the jemalloc metrics (exposed by jemalloc-ctl) to figure
out how much memory we are consuming; if we don't enable `unprefixed_malloc_on_supported_platforms` and only install the rust global allocator everything works just fine, except the memory metrics return a lower number because memory allocated by e.g. croaring bitmaps wouldn't be tracked.
However since heappy is is incompatible with `unprefixed_malloc_on_supported_platforms` but effectively plays the same role (by installing a replacement of malloc that ultimately invokes jemalloc) and we
were in a rush last week, we turned on heappy and turned off `unprefixed_malloc_on_supported_platforms`
without doing the full work of figuring out how to tell Cargo that those flags have that mutually-exclusive logic; the followup work has been put in a #2288.
This PR removes this rust global allocator since it's a source of confusion. Now if we'll accidentally
disable jemalloc for C we'll also disable it for rust which will be immediately visible since we won't
have any memory metrics at all.
Rename `frequency` to `interval` and set the default to `1`, which means that
every allocation will be counted.
Before this PR, we were incorrectly reusing the same "frequency" parameter (and related default of 99)
which meant that we were skipping small allocations that.
The sampling interval is a number of bytes that have to cumulatively allocated for a sample to be taken.
For example if the sampling interval is 99, and you're doing a million of 40 bytes allocations, the allocations profile will account for 16MB instead of 40MB. Heappy will adjust the estimate for sampled recordings, but now that feature is not yet implemented.
The object store handling is a large part of our CLI and kinda inlined
into the rest of the setup routines. This commit moves the object store
related code into its own module. As a bonus it can now be reused by
other CLI commands (e.g. low-level debug tools).
There are no functional changes here except for slight changes in the
error structure.
* fix: nocache feature code rot
The MBChunk::snapshot code when using the "nocache" option no longer
compiles - this commit updates it to match the not(nocache) code.
* build: use updated broken_intra_doc_links name
The broken_intra_doc_links lint was renamed
rustdoc::broken_intra_doc_links
https://doc.rust-lang.org/rustdoc/lints.html
* feat: drain database jobs on shutdown
* chore: fmt
* chore: review feedback
* chore: use join() not member directly
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Raphael Taylor-Davies
kodiakhq[bot]
Marco Neumann
Marko Mikulicic
Andrew Lamb
Jason
Nga Tran
Andrew Lamb
Dom