* fix(influxd): update xxhash, avoid stringtoslicebyte in cache (#578)
* fix(influxd): update xxhash, avoid stringtoslicebyte in cache
This commit does 3 things:
* it updates xxhash from v1 to v2; v2 includes a assembly arm version of
Sum64
* it changes the cache storer to write with a string key instead of a
byte slice. The cache only reads the key which WriteMulti already has
as a string so we can avoid a host of allocations when converting back
and forth from immutable strings to mutable byte slices. This includes
updating the cache ring and ring partition to write with a string key
* it updates the xxhash for finding the cache ring partition to use
Sum64String which uses unsafe pointers to directly use a string as a
byte slice since it only reads the string. Note: this now uses an
assembly version because of the v2 xxhash update. Go 1.22 included new
compiler ability to recognize calls of Method([]byte(myString)) and not
make a copy but from looking at the call sites, I'm not sure the
compiler would recognize it as the conversion to a byte slice was
happening several calls earlier.
That's what this change set does. If we are uncomfortable with any of
these, we can do fewer of them (for example, not upgrade xxhash; and/or
not use the specialized Sum64String, etc).
For the performance issue in maz-rr, I see converting string keys to
byte slices taking between 3-5% of cpu usage on both the primary and
secondary. So while this pr doesn't address directly the increased cpu
usage on the secondary, it makes cpu usage less on both which still
feels like a win. I believe these changes are easier to review that
switching to a byte slice pool that is likely needed in other places as
the compiler provides nearly all of the correctness checks we need (we
are relying also on xxhash v2 being correct).
* helps #550
* chore: fix tests/lint
* chore: don't use assembly version; should inline
This 2 line change causes xxhash to use a purego Sum64 implementation
which allows the compiler to see that Sum64 only read the byte slice
input which them means is can skip the string to byte slice allocation
and since it can skip that, it should inline all the calls to
getPartitionStringKey and Sum64 avoiding 1 call to Sum64String which
isn't inlined.
* chore: update ci build file
the ci build doesn't use the make file!!!
* chore: revert "chore: update ci build file"
This reverts commit 94be66fde03e0bbe18004aab25c0e19051406de2.
* chore: revert "chore: don't use assembly version; should inline"
This reverts commit 67d8d06c02e17e91ba643a2991e30a49308a5283.
(cherry picked from commit 1d334c679ca025645ed93518b7832ae676499cd2)
* feat: need to update go sum
---------
Co-authored-by: Phil Bracikowski <13472206+philjb@users.noreply.github.com>
(cherry picked from commit 06ab224516)
* fix(tsi1/partition/test): fix data races in test code (#57)
* fix(tsi1/partition/test): fix data races in test code
This PR is like influxdata/influxdb#24613 but solves it with a setter
method for MaxLogFileSize which allows unexporting that value and
MaxLogFileAge. There are actually two places locks were needed in test
code. The behavior of production code is unchanged.
(cherry picked from commit f0235c4daf4b97769db932f7346c1d3aecf57f8f)
* feat: modify error handling to be more idiomatic
closes https://github.com/influxdata/influxdb/issues/24042
* fix: errors.Join() filters nil errors
---------
Co-authored-by: Phil Bracikowski <13472206+philjb@users.noreply.github.com>
* chore: add scaffolding for naive solution
* feat: test case scaffolding
* fix: implement check for series key before proceeding
* fix: add validation for ReadSeriesKeyMeasurement usage
* refactor: explicit use of series key len
* feat: add remaining check to index
* feat: add check to remaining files
As the Len function is used as part of the parseSeriesKey, this also needs to be accounted for on the nil return from this function as it is used in different contexts
* feat: expand test cases
* chore: go fmt
* chore: update test failure message
* chore: impl feedback on unnecessary sz checks
* feat: expand test cases
* fix: nil series key check
In both sections for index.go there is a pre-existing length check against the series key which should catch invalid values, perhaps this explains why it hasn't cropped up in the reported panics. For even more safety, we can also skip a nil key because we know that subsequent calls will cause a panic where this key is attempted to be used
* fix: remove nil tags check
A key with no tags is valid, so we should not check for BOTH nil key and tags as a key could be nil, which is invalid, yet still have tags and therefore cause the check to pass which we do not want
* feat: extend test cases from feedback
* fix: extend checks for CompareSeriesKeys
* feat: add nilKeyHandler for shared key checking logic
* fix: logical error in nilKeyHandler
Prior to this, the else was always defaulted to at the end of the conditional branch, which causes unexpected behaviour and a failure of a bunch of tests.
* fix: return tags keep nil data
In a recent change to this, we agreed on a simple name == nil check for the actual data. As a follow on to this, I just realised that we don't actually want to nil back the tags, even if they're not checked, because having no tags is a valid input so we can simply return whatever we were passed unchanged.
* fix: use len == 0 for extra safety
* feat: extra test for blank series key
* chore: upgrade Go to 1.19.3
This re-runs ./generate.sh and ./checkfmt.sh to format and update
source code (this is primarily responsible for the huge diff.)
* fix: update tests to reflect sorting algorithm change
Do not update the `FileSet` or `activeLogFile` field in the in-memory
Partition structure if the Manifest file is not correctly saved to
the disk.
closes https://github.com/influxdata/influxdb/issues/23553
When a MANIFEST file is created in TSI, it
should be written to a temp file, then
atomically renamed, to avoid overwriting
the existing file only to fail on the
later write.
closes https://github.com/influxdata/influxdb/issues/23536
* fix: Change from RewriteExpr to PartitionExpr
Also remove some dead code
* feat: WITH KEY implementation
* feat: query rewriting for WITH KEY in SHOW TAG KEYS
This patch protects an internal map for concurrent use.
(*LogFile).Writes() method calls
(*LogFile).createMeasurementIfNotExists() which writes to a shared map.
(*LogFile).Writes() acquires a read-lock which leaves
createMeasurementIfNotExists() open to concurrent writes to its shared
map.
This commit adds the ExecEntries method to the *LogFile type so that we
can properly lock calls to (*LogFile).appendEntry() using defer.
(*LogFile).ExecEntries() is used to mostly replace the body of
(*LogFile).Writes() and incurs another function call since ExecEntries()
can't be inlined. Below is the output of build with "-m -m -m" gcflags:
./log_file.go:1076:6: cannot inline (*LogFile).ExecEntries: unhandled op DEFER
The performace impact of the additional function call should be
negligable and is outwieghed by the safety and simplicity of using
defer.
This commit quiets staticcheck's warnings about "unnecessary use of
fmt.Sprintf" and "unnecessary use of fmt.Sprint".
Prior to this commit we were wrapping simple constant strings without
any formatting verbs with fmt.Sprintf().
We were seing segfaults in Roaring bitmaps sometimes, under very
high load with networked drives. This may reduce risk of segfault by
forcing marshalling to copy the data.
* fix: access tsi active log file with READ lock
The activeLogFile pointer may be altered by other routine so the READ
lock is needed.
* Merge pull request #16384 from foobar/tsi-partition-lock
fix: access tsi active log file with READ lock
Co-authored-by: Tristan Su <sooqing@gmail.com>
Co-authored-by: David Norton <dgnorton@gmail.com>
Prior to this change, new series would be added to the series file
before checking the series cardinality limit. If the limit was exceeded,
the write was rejected even though the series had already been added to
the series file.
Fixes#15859
This commit fixes a defect in the TSI index where a filter using the
negated equality operator would result in no matching series being
returned for series stored within the `IndexFile` portions of the index.
The root cause of this was due to missing legacy-handling code in the
index for this particular iterator.
Fixes#10052
This commit fixes an issue where field keys would reappear in results
when querying previously dropped measurements.
The issue manifests itself when duplicates of a new series are inserted
into the `inmem` index. In this case, a map that tracks the number of
series belonging to a measurement was incorrectly incremented once for
each duplication of the series. Then, when it came time to drop the
measurement, the index assumed there were several series belonging to
the measurement left in the index (because the counter was higher than
it should be). The result of that was that the `fields.idx` file (which
stores a mapping between measurements and field keys) was not truncated
and rebuilt. This left old field keys in that file, which were then
returned in subsequent queries over all field keys.
Previously it was possible to set IDs on a `nil` entry which would
in turn cause a panic. If this panic was recovered by the server
then it would result in a mutex in the `inmem` index staying locked
indefinitely.
devanbenz
Shiwen Cheng
Jack
davidby-influx
Brandon Pfeifer
Dane Strandboge
lifeibo
Dane Strandboge
Sam Arnold
Tristan Su
Ayan George
dengzhi.ldz
Ben Johnson
docmerlin (j. Emrys Landivar)
Edd Robinson
David Norton
Jeff Wendling
Ben Wells