Avoid panicking when mapping points to a shard group
that has no shards. This does not address the root problem,
how the shard group ended up with no shards.
helps: https://github.com/influxdata/influxdb/issues/25715
* fix(influxd): update xxhash, avoid stringtoslicebyte in cache (#578)
* fix(influxd): update xxhash, avoid stringtoslicebyte in cache
This commit does 3 things:
* it updates xxhash from v1 to v2; v2 includes a assembly arm version of
Sum64
* it changes the cache storer to write with a string key instead of a
byte slice. The cache only reads the key which WriteMulti already has
as a string so we can avoid a host of allocations when converting back
and forth from immutable strings to mutable byte slices. This includes
updating the cache ring and ring partition to write with a string key
* it updates the xxhash for finding the cache ring partition to use
Sum64String which uses unsafe pointers to directly use a string as a
byte slice since it only reads the string. Note: this now uses an
assembly version because of the v2 xxhash update. Go 1.22 included new
compiler ability to recognize calls of Method([]byte(myString)) and not
make a copy but from looking at the call sites, I'm not sure the
compiler would recognize it as the conversion to a byte slice was
happening several calls earlier.
That's what this change set does. If we are uncomfortable with any of
these, we can do fewer of them (for example, not upgrade xxhash; and/or
not use the specialized Sum64String, etc).
For the performance issue in maz-rr, I see converting string keys to
byte slices taking between 3-5% of cpu usage on both the primary and
secondary. So while this pr doesn't address directly the increased cpu
usage on the secondary, it makes cpu usage less on both which still
feels like a win. I believe these changes are easier to review that
switching to a byte slice pool that is likely needed in other places as
the compiler provides nearly all of the correctness checks we need (we
are relying also on xxhash v2 being correct).
* helps #550
* chore: fix tests/lint
* chore: don't use assembly version; should inline
This 2 line change causes xxhash to use a purego Sum64 implementation
which allows the compiler to see that Sum64 only read the byte slice
input which them means is can skip the string to byte slice allocation
and since it can skip that, it should inline all the calls to
getPartitionStringKey and Sum64 avoiding 1 call to Sum64String which
isn't inlined.
* chore: update ci build file
the ci build doesn't use the make file!!!
* chore: revert "chore: update ci build file"
This reverts commit 94be66fde03e0bbe18004aab25c0e19051406de2.
* chore: revert "chore: don't use assembly version; should inline"
This reverts commit 67d8d06c02e17e91ba643a2991e30a49308a5283.
(cherry picked from commit 1d334c679ca025645ed93518b7832ae676499cd2)
* feat: need to update go sum
---------
Co-authored-by: Phil Bracikowski <13472206+philjb@users.noreply.github.com>
* chore: loadShards changes to more cleanly support 2.x feature (#25513)
* chore: move shardID parsing and shard filtering into walkShardsAndProcess
* chore: make it impossible to miss sending shardResponse or marking shard as complete
* chore: always count number of shards (preparation for 2.x related feature)
* chore: explicitly load series files and create indices serially
Explicitly load series files and create indices serially. Also
avoid passing them to work functions that don't need them.
* chore: rework loadShards for changes necessary to cancel loading process
* chore: comment improvements
* fix: fix race conditions in TestStore_StartupShardProgress and TestStore_BadShardLoading
* chore: avoid logging nil error
* chore: refactor shard loading and shard walking
Refactor loadShards and CreateShard to use a common shardLoader class that
makes thread-safety easier. Refactor walkShardsAndProcess into findShards.
* chore: improve comment
* chore: rename OpenShard to ReopenShard and implement with shardLoader
Rename Store.OpenShard to Store.ReopenShard and implement using a
shardLoader object. Changes to tests as necessary.
* chore: avoid resetting shard options and locking on Reopen
Avoid resetting shard options when reopening a shard.
Proper mutex locker in Shard.ReopenShard.
* chore: fix formatting issue
* chore: warn on mixed index types in Store.CreateShard
* chore: change from info to warn when invalid shard IDs found in path
* chore: use coarser locking in Store.ReopenShard
* chore: fix typo in comment
* chore: code simplification
(cherry picked from commit 0bc167bbd7)
* chore: fix logging issues in Store.loadShards
Fix reporting shards not opening correctly when they actually did.
Fix race condition with logging in loadShards.
(cherry picked from commit 65683bf166)
* chore: remove unnecessary fmt.Sprintf calls
Remove unnecessary fmt.Sprintf calls for static code checks in main-2.x.
(cherry picked from commit 8497fbf0af)
* chore: remove unnecessary blank identifier
* chore: remove unnecessary blank identifier
Add `--pid-file` option to write PID files on startup. The PID filename
is specified by the argument after `--pid-file`. If the PID file already exists, influxd will exit unless the `--overwrite-pid-file` flag is also used.
Example: `influxd --pid-file /var/lib/influxd/influxd.pid`
PID files are automatically removed when the influxd process is shutdown.
Closes: 25473
* feat: add option to flush WAL on shutdown
Add `--storage-wal-flush-on-shutdown` to flush WAL on database shutdown.
On successful shutdown, all WAL data will be committed to TSM files and the
WAL directories will not contain any .wal files.
Closes: #25422
* fix(tsi1/partition/test): fix data races in test code (#57)
* fix(tsi1/partition/test): fix data races in test code
This PR is like influxdata/influxdb#24613 but solves it with a setter
method for MaxLogFileSize which allows unexporting that value and
MaxLogFileAge. There are actually two places locks were needed in test
code. The behavior of production code is unchanged.
(cherry picked from commit f0235c4daf4b97769db932f7346c1d3aecf57f8f)
* feat: modify error handling to be more idiomatic
closes https://github.com/influxdata/influxdb/issues/24042
* fix: errors.Join() filters nil errors
closes https://github.com/influxdata/influxdb/issues/25341
---------
Co-authored-by: Phil Bracikowski <13472206+philjb@users.noreply.github.com>
(cherry picked from commit 5c9e45f033)
If NewTSMReader() fails because mmap fails, do not
rename the file, because the error is probably
caused by vm.max_map_count being too low
Closes: #25337
(cherry picked from commit ec412f793b)
* fix(tsi1/partition/test): fix data races in test code
This PR is like #24613 but solves it with a setter
method for MaxLogFileSize which allows unexporting that value and
MaxLogFileAge. There are actually two places locks were needed in test
code. The behavior of production code is unchanged.
(cherry picked from commit f0235c4daf4b97769db932f7346c1d3aecf57f8f)
When querying data before 1970-01-01 (UNIX time 0)
validateArgs would set start to -in64 max and end to int64 max.
closes https://github.com/influxdata/influxdb/issues/24669
Co-authored-by: Paul Hegenberg <paul.hegenberg@gmail.com>
Avoid converting times to int64 in the Task Scheduler
to preserve time zone information. This corrects a
failure after fall back time changes which halts
every-type tasks
closes https://github.com/influxdata/influxdb/issues/25110
* fix: prevent retention service from hanging (#25055)
Fix issue that can cause the retention service to hang waiting on a
`Shard.Close` call. When this occurs, no other shards will be deleted
by the retention service. This is usually noticed as an increase in
disk usage because old shards are not cleaned up.
The fix adds to new methods to `Store`, `SetShardNewReadersBlocked`
and `InUse`. `InUse` can be used to poll if a shard has active readers,
which the retention service uses to skip over in-use shards to prevent
the service from hanging. `SetShardNewReadersBlocked` determines if
new read access may be granted to a shard. This is required to prevent
race conditions around the use of `InUse` and the deletion of shards.
If the retention service skips over a shard because it is in-use, the
shard will be checked again the next time the retention service is run.
It can be deleted on subsequent checks if it is no longer in-use. If
the shards is stuck in-use, the retention service will not be able to
delete the shards, which can be observed in the logs for manual
intervention. Other shards can still be deleted by the retention service
even if a shard is stuck with readers.
This is a port of ad68ec8 from master-1.x to main-2.x.
closes: #25076
(cherry picked from commit b4bd607eef)
Stacks and templates allow specifying file:// URLs. Add command line
option `--template-file-urls-disabled` to disable their use for people who don't require them.
* feat: update flux to latest head
Flux has updated some dependencies, including prometheus. Prometheus
has changed in some incompatible ways. Update the flux dependency
to a newer version with the updated prometheus dependency and apply
some small fixes to make everything build. This is in preparation
for a flux release later in the week.
The biggest change is in some tests that were using runtime.DeepEqual
to check the correctness of prometheus metrics. The internals of
these types have changed such that this is not a safe thing to do
anymore. The test now verifies the string representations, as
produced by String(), match.
* fix: update CI script
The scripts/ci/check-system-go-matches-go-mod.sh is failing because
newer go toolchains include the bugfix version in go.mod's go
directive. Update the script to check the major and minor versions
reported by both tools match.
This adds locking to the load method and renames it to Reload(). This
method replaces the cached data from the underlying kv.Store and needs a
write lock. The restore api uses it and may have been an issue with
concurrent writes into the cached data during a restore.
* fixes#24895
devanbenz
Jamie Strandboge
davidby-influx
Geoffrey Wossum
Martin Hilton
peterbarnett03
Brandon Pfeifer
Phil Bracikowski
Jakub Bednář