* Update the store to remove the WAL directories associated with a shard or database when they are deleted.
* Fix the Store so that it creates separate WAL directories for databases and retention policies.
This func show up in profiling. It's called frequently from multiple places and
can be made more efficient. The previous implementation looped over the input
slice 4 times updating an returning a new slice each time. The changes it to loop
once and create one result slice.
With influx_stress
Before:
Wrote 10000000 points at average rate of 241750
Average response time: 187.78968ms
After:
Wrote 10000000 points at average rate of 254618
Average response time: 172.235028ms
Through profiling of writes, point.Fields() and point.Name() were called
repeatedly in PointsWriter and the Shard. These calls are somewhat expensive
when writing large batches so we can cache them to avoid wasting CPU cycles.
Using influx_stress with default settings
Before:
Wrote 10000000 points at average rate of 202570
Average response time: 235.450355ms
After:
Wrote 10000000 points at average rate of 246120
Average response time: 182.881008ms
This commit changes the bz1 append to check for a small
ending block first. If the block is below the threshold
for block size then it is rewritten with the new data
points instead of having a new block written.
If a flush is happening and you bring up a cursor for a series, if that series didn't have any data in the cache (after the flush started) then it would return no data. What it should have done instead is return the data that is in the flush cache, which is held in separate area of memory until it is committed to the index.
If the measurement started with a quote, a panic would happen. This
is a reegression due to cb7f0b8.
This also uncovered that measurement names were being escaped incorrectly.
The escape codes for tag and fields also includes `=` and '"` which should
not be escaped for measurement names.
Fixes#3681
* All metadata for each shard is now stored in a single key with compressed value
* Creation of new metadata no longer requires a syncrhnous write to Bolt. It is passed to the WAL and written to Bolt periodically outside the write path
* Added DeleteSeries to WAL and updated bz1 to remove series there when DeleteSeries or DropMeasurement are called
CPU profiling shows that computing the tagset-based key of each point is significant CPU cost. These keys actually don't change per cursor, so precompute once at mapper-open time, and then use those values as points are drained from the cursor.
Before this change the cursor tag was getting computed on every point, which involved marshalling tags.
The series map on Measurement was updated and deleted from but never
actually used. Series keys can be very bia since they are the the
string representation of the measurement plus sorted tags.
Locally I see 20%-30% reduction in memory usage with 1M series.
This commit changes FieldCodec to always be non-nil. Normally it should
always be non-nil, however, if metadata is not persisted correctly or
consistently then it could be missing. A nil FieldCodec causes queries
to panic.
Fixes#3535
* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
This commit fixes the b1 cursor so that reads from either the cache
or bolt buffer will check against the previously read key to ensure
that two of the same keys are not returned.
Fixes#3571.
ValidateGroupBy was returning an error if a tag does not exist
but it appears that function was supposed to be validating that
a field name was not used as a group by field.
Fixes#3326
This commit adds a cursor that wraps multiple `tsdb.Cursor` objects
and streams them out as one cursor. The multi-cursor automatically
dedupes keys by using the first cursor specified in the argument
list.
This commit fixes issues found from using a more complex `testing/quick`
implementation of the `WriteIndex()` test. The newer test inserts
multiple sets of random data that's confined to a smaller random space
so there's more chance of overlapping data.
The fixes were primarily around inserting old data or inserting the same
timestamp multiple times for a single write. The block splitting was not
working correctly before and the sorting and deduping was not handled
correctly.
Newlines in a string field would cause the parser to return
the line prematurely causing "unbalanced quotes" errors. This
makes the line scanning aware of quote fields so that the whole
line is returned.
Fixes#3545
This change moves tracking of next timestamp and values to simple
slices, as performance measurement showed that Peek() on TagSet cursors
was a huge performance drain. There is much more that can be done here,
but with this in place query performance has been restored to 0.9.1
levels.
This change also uses -1 to indicate that no value is available for a
given timestamp.
With this change remote mapping no longer uses HTTP, as the HTTP ports
exposed by nodes on the cluster are not known cluster wide. The TCP
ports exposed by the cluster service are, so this change uses that
functionality. Each RemoteMapper has its own dedicated connection pool
for each node, and remote mapping TCP connections are in no way coupled
with query TCP connections.
This change significantly simplifies query executor code. Before this
change there were two types of executors -- RawExecutor and
AggregateExecutor. These two types only differed in one function
Execute(). Otherwise all other methods on the Executors were common and
duplicated between executors
This change merges the two executors into a single type called, wait for
it, Executor and simply switches execute functions depending on the
statement type.
The multiple checks for Mapper and Executor type -- the lack of DRYness
in this code -- meant the same checks would need to be copied. Therefore
this change, as well as fixing the bug, improves the situation a little
bit by *asking* the Mappers what type of Executor is required. This code
is still not ideal.
Fixes#3355.
With this change, the query engine code gathers information about
shards and tagsets by working with individual shards, collating the
information, and returning that to the client. It does not assume that any
particular shard is local, and accesses all shards through abstracted
Mappers, of which there are two types -- a Mapper type for Raw queries
and a second type for Aggregate queries. There are corresponding
Executors for each type of Mapper, but both types of Executors share the
same interface.
Writing points that were not sorted by time could cause very high
CPU usages and increased latencies because each point inserted would
cause the in-memory cache to be resorted. The worst case would be
writing a large batch of N points in reverse time order which would
invoke N sorts of the slice.
This patch keeps track of which slices need to be sorted and sorts
them once at the end. In the previous example, the N sorts becomes
one. There is still a pathalogical case that would require N/2 sorts.
For example, 10000 points split across 5000 series. Each series has two
points that are in reverse time order. This would incur 5000 sorts still.
Fixes#3159
These are not supported types but previously it would cause the
point.Fields() func to panic. This prevents it from panicing
so the values can be ignored if needed.
When creating a point manually, the field values are interface{}
which allows unsupported types to be passed in. Previously, the
code would panic. It will now default to string representation of
the value if it's not a known type.
This commit adds a write ahead log to the shard. Entries are cached
in memory and periodically flushed back into the index. The WAL and
the cache are both partitioned into buckets so that flushing doesn't
stop the world as long.
Field values that were out of range for the type would panic the database
when being inserted because the parser would allow them as valid points.
This change prevents those invalid values from being parsed and instead
returns an error.
An alternative fix considered was to handle the error and clamp the value
to the min/max value for the type. This would treat numeric range errors
slightly differently than other type erros which might lead to confusion.
The simplest fix with the current parser would be to just convert each field
to the type at parse time. Unfortunately, this adds extra memory allocations
and lowers throughput significantly. Since out of range values are less common
than in-range values, some heuristics are used to determine when the more
expensive type parsing and range checking is performed. Essentially, we only
do the slow path when we cannot determine that the value is in an acceptable
type range.
Fixes#3127
A field value of just a numeric value would be accepted by the line
protocol parser but the value would be set as the field name and
the value would be nil. Instead, return an error because all field
values need a field name.
Statements were only being normalized if a default database was included
in the query (usually via the query param 'db'). However if no default
database was included, and none was an explicit part of the measurement
name, no database-existence check was run. This result in a later panic
with wildcard expansion.
Fixes#2960
Integers were were written back to line protocol using strconv.FormatFloat
incorrectly. Large integers are written in scientific notation which
causes their type to change to a float when parsed back.
Paul Dix
Jason Wilder
Daniel Morsing
Ben Johnson
Philip O'Toole
Philip O'Toole
Paul Dix
Cory LaNou
Cory LaNou
Michael Desa
David Norton
Josh Horwitz
gunnaraasen
Alexander Morozov
Joseph Crail
Sean Beckett
Todd Persen
ben hockey