This change delays Tag cloning until a new series is found, and will
only clone Tags acquired from `ParsePoints...` and not those referencing
the mmap-ed files (TSM) that are created on startup.
This leak seems to have been introduced in 8aa224b22d,
present in 1.1.0 and 1.1.1.
When points were parsed from HTTP payloads, their tags and fields
referred to subslices of the request body; if any tag set introduced a
new series, then those tags then were stored in the in-memory series
index objects, preventing the HTTP body from being garbage collected. If
there were no new series in the payload, then the request body would be
garbage collected as usual.
Now, we clone the tags before we store them in the index. This is an
imperfect fix because the Point still holds references to the original
tags, and the Point's field iterator also refers to the payload buffer.
However, the current write code path does not retain references to the
Point or its fields; and this change will likely be obsoleted when TSI
is introduced.
This change likely fixes#7827, #7810, #7778, and perhaps others.
I haven't been able to reproduce creating a point without any fields,
but we've seen points in the wild that have been marshalled with no
fields - that is, the length header for fields is uint32(0) and a
well-formed encoded time follows.
Attempting to unmarshal points via NewPointFromBytes returns
ErrPointMustHaveAField, so it seems better to fail earlier with the same
error, rather than allowing those points to be serialized in the first
place.
A string field w/ a trailing slash before the quote would parse incorrectly
because the quote would be seen as escaped. We have to treat \\ as an
escape sequence within strings in order to handle this.
The FieldIterator is used to scan over the fields of a point, providing
information, and delaying parsing/decoding the value until it is needed.
This change uses this new type to avoid the allocation of a map for the
fields which is then thrown away as soon as the points get converted
into columns within the datastore.
+ Remove a heap alloc in (Point).HashID() and (Row).tagsHash()
(According to `-gcflags -m`).
+ Direct port from the stdlib.
+ Fuzz test for equivalence to stdlib version.
+ Save one alloc per line when writing with the bulk protocol.
Over a longer period of writes, this allocation shows up quite
a bit in profiles since the slice needs to be resized frequently.
This scans the slice to count how many lines are going to be parsed
in order to pre-allocate the slice capacity. It's slightly slower,
but creates less garbage in the long run.
The v2 UDP client will attempt to split points that exceed the
configured payload size. It will only do this for points that have a
timestamp specified.
Truncate the time interval output of the monitor service to be on even
time intervals rather than on every minute based on the start time. This
normalizes the output from the monitor service.
The tsdb package had a substantial amount of dead code related to the
old query engine still in there. It is no longer used, so it was removed
since it was left unmaintained. There is likely still more code that is
the same, but wasn't found as part of this code cleanup.
influxql has dead code show up because of the code generation so it is
not included in this pruning.
The code for parsing a key our of the WAL or TSM files in the engine
was naive and didn't account for measurements with escape chars. This
uses the correct parsing code to parse and load them correctly.
Fixes#6496
Influx does not support fields with empty names or points
with no fields.
NewPoint is changed to validate that all field names are non-empty.
AddField is removed because we now require that all fields are
specified on construction.
NewPointFromByte is changed to return an error if a unmarshaled
binary point does not have any fields.
newFieldsFromBinary is changed to prevent an infinite loop that
can arise while attempting to parse corrupt binary point data.
TestNewPointsWithBytesWithCorruptData is changed to reflect the
change in the behaviour of NewPointFromByte.
Signed-off-by: Jon Seymour <jon@wildducktheories.com>
byte.Replace will return a copy of the input even when nothing has
been replaced. This is called in lower level query execution and
create some garbage that isn't necessary.
Quotes are handled differently in the line protocol depending on when
they are encountered. Quotes in field values matter, quotes anywhere
else don't.
`scanLine()` didn't understand this difference and treated all quotes
the same as ones for tag values. This resulted in `scanLine()` reading
the wrong amount of data sometimes when quotes were involved.
This fixes#5204.
Quotes are not supposed to be significant in field keys, but are
significant in field values. The code as it currently was would
consider quotes in a key to be significant, but the later parser that
would unmarshal the fields from the byte string did not consider those
quotes to be significant. This meant that the following string:
"a=1
The line protocol parser would see a mismatched quote instead of a valid
input to the line protocol. But more nefariously, the following string:
"a=1"=2
The line protocol parser would ignore the first equals since it is
located in the quotation marks and think this was a valid input. It
would then pass it on to the field parser who would panic and die when
it tried to parse `1"=2` as a number.
Fixes#4076.
Mainly for debugging as since this should not happen going forward. Since
there may be points with NaN already stored in the WAL, this is helpful for
troubleshooting panics.
Float values are not supported in the existing engine and the tsm1
engines. This changes NewPoint to return an error if a field value
contains a NaN field. It also allows us to validate fields to prevent
other unsupported types from sneaking in through other input plugins.
Edd Robinson
Ben Johnson
Joe LeGasse
Mark Rushakoff
Mark Rushakoff
Cory LaNou
kun
Jason Wilder
rw
Vladimir Sagan
Jonathan A. Sternberg
Jon Seymour
Justin Nuß
Nathaniel Cook
CrazyJvm
Cameron Sparr
Jason Wilder
ch33hau
gunnaraasen
Charles Chan
Sean Beckett
Philip O'Toole
Paul Dix
Paris Holley