* fix(influxd): update xxhash, avoid stringtoslicebyte in cache (#578)
* fix(influxd): update xxhash, avoid stringtoslicebyte in cache
This commit does 3 things:
* it updates xxhash from v1 to v2; v2 includes a assembly arm version of
Sum64
* it changes the cache storer to write with a string key instead of a
byte slice. The cache only reads the key which WriteMulti already has
as a string so we can avoid a host of allocations when converting back
and forth from immutable strings to mutable byte slices. This includes
updating the cache ring and ring partition to write with a string key
* it updates the xxhash for finding the cache ring partition to use
Sum64String which uses unsafe pointers to directly use a string as a
byte slice since it only reads the string. Note: this now uses an
assembly version because of the v2 xxhash update. Go 1.22 included new
compiler ability to recognize calls of Method([]byte(myString)) and not
make a copy but from looking at the call sites, I'm not sure the
compiler would recognize it as the conversion to a byte slice was
happening several calls earlier.
That's what this change set does. If we are uncomfortable with any of
these, we can do fewer of them (for example, not upgrade xxhash; and/or
not use the specialized Sum64String, etc).
For the performance issue in maz-rr, I see converting string keys to
byte slices taking between 3-5% of cpu usage on both the primary and
secondary. So while this pr doesn't address directly the increased cpu
usage on the secondary, it makes cpu usage less on both which still
feels like a win. I believe these changes are easier to review that
switching to a byte slice pool that is likely needed in other places as
the compiler provides nearly all of the correctness checks we need (we
are relying also on xxhash v2 being correct).
* helps #550
* chore: fix tests/lint
* chore: don't use assembly version; should inline
This 2 line change causes xxhash to use a purego Sum64 implementation
which allows the compiler to see that Sum64 only read the byte slice
input which them means is can skip the string to byte slice allocation
and since it can skip that, it should inline all the calls to
getPartitionStringKey and Sum64 avoiding 1 call to Sum64String which
isn't inlined.
* chore: update ci build file
the ci build doesn't use the make file!!!
* chore: revert "chore: update ci build file"
This reverts commit 94be66fde03e0bbe18004aab25c0e19051406de2.
* chore: revert "chore: don't use assembly version; should inline"
This reverts commit 67d8d06c02e17e91ba643a2991e30a49308a5283.
(cherry picked from commit 1d334c679ca025645ed93518b7832ae676499cd2)
* feat: need to update go sum
---------
Co-authored-by: Phil Bracikowski <13472206+philjb@users.noreply.github.com>
* feat: update flux to latest head
Flux has updated some dependencies, including prometheus. Prometheus
has changed in some incompatible ways. Update the flux dependency
to a newer version with the updated prometheus dependency and apply
some small fixes to make everything build. This is in preparation
for a flux release later in the week.
The biggest change is in some tests that were using runtime.DeepEqual
to check the correctness of prometheus metrics. The internals of
these types have changed such that this is not a safe thing to do
anymore. The test now verifies the string representations, as
produced by String(), match.
* fix: update CI script
The scripts/ci/check-system-go-matches-go-mod.sh is failing because
newer go toolchains include the bugfix version in go.mod's go
directive. Update the script to check the major and minor versions
reported by both tools match.
Under certain circumstances, the retention service can fail to delete shards from
the store in a timely manner. When the shard groups are pruned based on age, this
leaves orphaned shard files on the disk. The retention service will then not attempt
to remove the obsolete shard files because the meta store does not know about them.
This can cause excessive disk space usage for some users.
This corrects that by requiring shards files be deleted before they can be removed
from the meta store.
fixes: #24529
(cherry picked from commit 7bd3f89d18)
closes https://github.com/influxdata/influxdb/issues/24545
Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com>
* chore: bump testcontainers to v0.15.0
* chore: run go mod tidy
* chore: update test to latest version of testcontainers
* chore: update package
* fix: use collectors.NewGoCollector instead
SA1019 detected by staticcheck
* build(flux): update flux to v0.173.0
* fix(fluxfmt): update tests for newline in Flux fmt
Flux fmt now explicitly adds a newline to the end of a file, updating
tests accordingly.
WeblWabl
Martin Hilton
Jamie Strandboge
davidby-influx
Brandon Pfeifer
Christopher M. Wolff
dependabot[bot]
fuyou
Manuel de la Peña
Jeffrey Smith II
Chunchun Ye
Manuel de la Peña
Nathaniel Cook
Dane Strandboge
Sunil Kartikey
Sean Brickley
Jonathan A. Sternberg