* the protocol service definition, ReadRequest and ReadResponse is
reused across projects, rather than requiring redefinition.
* the ReadRequest protocol buffer definition removes the concept of a
database and retention policy, replacing it with a field named
ReadSource of type google.protobuf.Any. OSS requests will use the
ReadSource message structure defined in local to this package, which
defines fields to represent a Database and RetentionPolicy. Other
implementations can provide their own data structure allowing the
remainder of the ReadRequest to be reused.
* The RPC service and Store are expected to be redefined to handle their
specific requirements for resolving a ReadSource
* ResultSet and GroupResultSet are interfaces representing non-grouping
and grouping read behavior respectively. Calling NewResultSet or
NewGroupResultSet will construct instances of these types
* The ResponseWriter type is exported to deal with serialization of
the ResultSet and GroupResultSet types
Since we append to the file itself, once we have read the file in, we
can be done with the mmap'd data.
Ideally we can rework UnmarshalBinary and do away with the mmap
completely. That is future work.
This commit ensures that any orphaned series (series that are to be
removed and no longer are referenced anywhere in the database) are
removed from the `inmem` index when a shard is dropped.
it's slightly slower, but the safety is worth it i think.
```
name old time/op new time/op delta
Next-8 30.0ns ± 2% 31.0ns ± 3% +3.56% (p=0.002 n=7+8)
NextParallel-8 79.4ns ± 1% 92.5ns ± 1% +16.58% (p=0.000 n=8+8)
```
use atomics rather than mutexes to synchronize state between calls.
```
name old time/op new time/op delta
Next-8 244ns ± 0% 30ns ± 2% -87.70% (p=0.000 n=8+7)
NextParallel-8 215µs ±60% 0µs ± 1% -99.96% (p=0.000 n=8+8)
```
The results for NextParallel are around ~80ns/op, but that doesn't
show up in the benchstat output.
Since all tag sets are materialised to strings before this method
returns, a large number of allocations can be avoided by carefully
resuing buffers and containers.
This commit reduces allocations by about 75%, which can be very
significant for high cardinality workloads.
The benchmark results shown below are for a benchmark that asks for all
series keys matching `tag5=value0'.
name old time/op new time/op delta
Index_ConcurrentWriteQuery/inmem/queries_100000-8 5.66s ± 4% 5.70s ± 5% ~ (p=0.739 n=10+10)
Index_ConcurrentWriteQuery/tsi1/queries_100000-8 26.5s ± 8% 26.8s ±12% ~ (p=0.579 n=10+10)
IndexSet_TagSets/1M_series/inmem-8 11.9ms ±18% 10.4ms ± 2% -12.81% (p=0.000 n=10+10)
IndexSet_TagSets/1M_series/tsi1-8 23.4ms ± 5% 18.9ms ± 1% -19.07% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
Index_ConcurrentWriteQuery/inmem/queries_100000-8 2.50GB ± 0% 2.50GB ± 0% ~ (p=0.315 n=10+10)
Index_ConcurrentWriteQuery/tsi1/queries_100000-8 32.6GB ± 0% 32.6GB ± 0% ~ (p=0.247 n=10+10)
IndexSet_TagSets/1M_series/inmem-8 3.56MB ± 0% 3.56MB ± 0% ~ (all equal)
IndexSet_TagSets/1M_series/tsi1-8 12.7MB ± 0% 5.2MB ± 0% -59.02% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
Index_ConcurrentWriteQuery/inmem/queries_100000-8 24.0M ± 0% 24.0M ± 0% ~ (p=0.353 n=10+10)
Index_ConcurrentWriteQuery/tsi1/queries_100000-8 96.6M ± 0% 96.7M ± 0% ~ (p=0.579 n=10+10)
IndexSet_TagSets/1M_series/inmem-8 51.0 ± 0% 51.0 ± 0% ~ (all equal)
IndexSet_TagSets/1M_series/tsi1-8 80.4k ± 0% 20.4k ± 0% -74.65% (p=0.000 n=10+10)
The internals of `newSeriesCursor` returned a struct pointer that
implicitly got turned into the interface. Unfortunately, Go treats this
type of interface conversion as a nil pointer to the struct rather than
as just nil so if you attempted to compare the returned cursor to nil,
they would not be equal and it would think it was non-nil and attempt to
use the cursor.
Jeff Wendling
Stuart Carnie
Edd Robinson
Hercules Mango Churchill
David Norton
Jeff Wendling
tmgordeeva
Ben Johnson
liukun
Tim Brooks
Jonathan A. Sternberg
Jacob Marble