Start of a lower-level file inspection tool. This currently dumps
summary statistics for the shards, index and WAL that can be used to
understand the shape of the data is in the local shards. This util
operates on the shards itself and not through the server and is intended
more for debugging/troubleshooting.
Writes could timeout and when adding new measurement names to the
index if the sort took a long time. The names slice was never
actually used (except a test) so keeping it in index wastes memory
and sort it wastes CPU and increases lock contention. The sorting
was happening while the shard held a write-lock.
Fixes#3869
* All metadata for each shard is now stored in a single key with compressed value
* Creation of new metadata no longer requires a syncrhnous write to Bolt. It is passed to the WAL and written to Bolt periodically outside the write path
* Added DeleteSeries to WAL and updated bz1 to remove series there when DeleteSeries or DropMeasurement are called
The series map on Measurement was updated and deleted from but never
actually used. Series keys can be very bia since they are the the
string representation of the measurement plus sorted tags.
Locally I see 20%-30% reduction in memory usage with 1M series.
ValidateGroupBy was returning an error if a tag does not exist
but it appears that function was supposed to be validating that
a field name was not used as a group by field.
Fixes#3326
With this change, the query engine code gathers information about
shards and tagsets by working with individual shards, collating the
information, and returning that to the client. It does not assume that any
particular shard is local, and accesses all shards through abstracted
Mappers, of which there are two types -- a Mapper type for Raw queries
and a second type for Aggregate queries. There are corresponding
Executors for each type of Mapper, but both types of Executors share the
same interface.
* Add deleteMeasurement to store and shard
* Add DropMeasurement to DatabaseIndex
* Update ErrMeasurementNotFound and ErrDatabaseNotFound to not include the first line of the stack trace.
* Pulled over updates to ast and parser from master
* Updated store and shard to be able to drop series
* Pulled updates to database.go from master into tsdb/meta.go
Sean Beckett
David Norton
Ben Johnson
Mark Rushakoff
Cory LaNou
Jason Wilder
Paul Dix
Philip O'Toole
Philip O'Toole
Pradeep Chhetri