The TagValues API will perform a linear scan if there is no predicate;
otherwise, it will use the index to find a list of candidate series
keys.
TagValues expects the predicate to be transformed such that
`_measurement` and `_field` are remapped to `\x00` and `\xff`
respectively.
There is one TODO marked to analyze the predicate for a
`\x00 = '<measurement>'` pattern. If found, the predicate can be
eliminated and fall back to a linear prefix scan by combining the org,
bucket and measurement.
This commit adds a new Cache option, via the
`tsm1.CacheConfig.SnapshotAgeDuration` field, which controls the maximum
age the cache can reach before it is snapshotted to a TSM file.
The default value for this option is `0`, which means that the cache
will never be snapshotted based only on age. Setting this value to, for
example, 10 seconds, would result in the cache snapshotting every 10
seconds.
Snapshotting the cache more frequently can provide better durability
guarantees in some circumstances, though more, smaller TSM files will
lead to more work needed to compact them down to larger, more dense
files.
When using InfluxDB with a WAL there isn't really a strong reason to
alter `tsm1.CacheConfig.SnapshotAgeDuration` from `0`.
This commit adds the pkg/lifecycle.Resource to help manage opening,
closing, and leasing out references to some resource. A resource
cannot be closed until all acquired references have been released.
If the debug_ref tag is enabled, all resource acquisitions keep
track of the stack trace that created them and have a finalizer
associated with them to print on stderr if they are leaked. It also
registers a handler on SIGUSR2 to dump all of the currently live
resources.
Having resources tracked in a uniform way with a data type allows us
to do more sophisticated tracking with the debug_ref tag, as well.
For example, we could panic the process if a resource cannot be
closed within a certain time frame, or attempt to figure out the
DAG of resource ownership dynamically.
This commit also fixes many issues around resources, correctness
during error scenarios, reporting of errors, idempotency of
close, tracking of memory for some data structures, resource leaks
in tests, and out of order dependency closes in tests.
It turns out that LastModified and DiskSize are unused, and so it
was easy to change to not care about the WAL.
This hooks up metrics and starts the WAL again.
I did this with a dumb editor macro, so some comments changed too.
Also rename root package from platform to influxdb.
In interest of minimizing risk, anyone importing the root package has
now aliased it to "platform" so that no changes beyond imports were
necessary in those files.
Lastly, replace the old platform module to local path /dev/null so that
nobody can accidentally reintroduce a platform dependency while
migrating platform code to influxdb.
The tsdb/tsm1 package was one of the test suites that took the longest
to run in platform with go test -short. The rule of thumb on the Go
project is that short mode should skip any individual test that takes
longer than one second. This change skips two such tests, and it
eliminates a string concatenation loop in two other tests, so that they
report completion in "0.00s" rather than about 0.94s, on my machine.
These cumulative changes take `go test -short ./tsdb/tsm1` from about 14
seconds to about 7 seconds on my machine.
- Add some documentation.
- Move compaction planner to an option instead of config.
The latter fits with the general theme of having config be things
that can be specified in a toml, and everything else being an
option.
This commit removes the remaining bits of the fields index. In doing
so, the buildCursor method on the engine would need to be updated.
It turns out, that code was statically dead, so delete it and anything
that depended on it. Additionally, delete anything as reported by
the unused tool in the tsdb package.
We are moving the necessary code for 2.0 from the influxdb 1.X
repository to the platform 2.0 repository. The logger is an unnecessary
dependency on the old influxdb that is making life more complicated.
Edd Robinson
tmgordeeva
Jeff Wendling
Ben Johnson
Stuart Carnie
Jacob Marble
Mark Rushakoff
zhulongcheng
Mark Rushakoff
Jeff Wendling
Jonathan A. Sternberg