Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/storage/reads/group_resultset_test.go

343 lines
8.8 KiB
Go
Raw Normal View History

chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
package reads_test
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
import (
"context"
"strings"
"testing"
"github.com/google/go-cmp/cmp"
"github.com/influxdata/platform/models"
chore(gen): Add ingen generator data structures to platform for reuse
2018-12-10 20:53:16 +00:00
"github.com/influxdata/platform/pkg/data/gen"
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
"github.com/influxdata/platform/storage/reads"
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
"github.com/influxdata/platform/storage/reads/datatypes"
)
func TestGroupGroupResultSetSorting(t *testing.T) {
tests := []struct {
name string
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
cur reads.SeriesCursor
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
group datatypes.ReadRequest_Group
keys []string
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
exp string
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
}{
{
name: "group by tag1 in all series",
cur: &sliceSeriesCursor{
rows: newSeriesRows(
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
"cpu,tag0=val00,tag1=val10",
"cpu,tag0=val00,tag1=val11",
"cpu,tag0=val00,tag1=val12",
"cpu,tag0=val01,tag1=val10",
"cpu,tag0=val01,tag1=val11",
"cpu,tag0=val01,tag1=val12",
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
)},
group: datatypes.GroupBy,
keys: []string{"tag1"},
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
exp: `group:
tag key : _m,tag0,tag1
partition key: val10
series: _m=cpu,tag0=val00,tag1=val10
series: _m=cpu,tag0=val01,tag1=val10
group:
tag key : _m,tag0,tag1
partition key: val11
series: _m=cpu,tag0=val00,tag1=val11
series: _m=cpu,tag0=val01,tag1=val11
group:
tag key : _m,tag0,tag1
partition key: val12
series: _m=cpu,tag0=val00,tag1=val12
series: _m=cpu,tag0=val01,tag1=val12
`,
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
},
{
name: "group by tag1 in partial series",
cur: &sliceSeriesCursor{
rows: newSeriesRows(
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
"aaa,tag0=val00",
"aaa,tag0=val01",
"cpu,tag0=val00,tag1=val10",
"cpu,tag0=val00,tag1=val11",
"cpu,tag0=val00,tag1=val12",
"cpu,tag0=val01,tag1=val10",
"cpu,tag0=val01,tag1=val11",
"cpu,tag0=val01,tag1=val12",
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
)},
group: datatypes.GroupBy,
keys: []string{"tag1"},
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
exp: `group:
tag key : _m,tag0,tag1
partition key: val10
series: _m=cpu,tag0=val00,tag1=val10
series: _m=cpu,tag0=val01,tag1=val10
group:
tag key : _m,tag0,tag1
partition key: val11
series: _m=cpu,tag0=val01,tag1=val11
series: _m=cpu,tag0=val00,tag1=val11
group:
tag key : _m,tag0,tag1
partition key: val12
series: _m=cpu,tag0=val01,tag1=val12
series: _m=cpu,tag0=val00,tag1=val12
group:
tag key : _m,tag0
partition key: <nil>
series: _m=aaa,tag0=val00
series: _m=aaa,tag0=val01
`,
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
},
{
name: "group by tag2,tag1 with partial series",
cur: &sliceSeriesCursor{
rows: newSeriesRows(
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
"aaa,tag0=val00",
"aaa,tag0=val01",
"cpu,tag0=val00,tag1=val10",
"cpu,tag0=val00,tag1=val11",
"cpu,tag0=val00,tag1=val12",
"mem,tag1=val10,tag2=val20",
"mem,tag1=val11,tag2=val20",
"mem,tag1=val11,tag2=val21",
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
)},
group: datatypes.GroupBy,
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
keys: []string{"tag2", "tag1"},
exp: `group:
tag key : _m,tag1,tag2
partition key: val20,val10
series: _m=mem,tag1=val10,tag2=val20
group:
tag key : _m,tag1,tag2
partition key: val20,val11
series: _m=mem,tag1=val11,tag2=val20
group:
tag key : _m,tag1,tag2
partition key: val21,val11
series: _m=mem,tag1=val11,tag2=val21
group:
tag key : _m,tag0,tag1
partition key: <nil>,val10
series: _m=cpu,tag0=val00,tag1=val10
group:
tag key : _m,tag0,tag1
partition key: <nil>,val11
series: _m=cpu,tag0=val00,tag1=val11
group:
tag key : _m,tag0,tag1
partition key: <nil>,val12
series: _m=cpu,tag0=val00,tag1=val12
group:
tag key : _m,tag0
partition key: <nil>,<nil>
series: _m=aaa,tag0=val00
series: _m=aaa,tag0=val01
`,
},
{
name: "group by tag0,tag2 with partial series",
cur: &sliceSeriesCursor{
rows: newSeriesRows(
"aaa,tag0=val00",
"aaa,tag0=val01",
"cpu,tag0=val00,tag1=val10",
"cpu,tag0=val00,tag1=val11",
"cpu,tag0=val00,tag1=val12",
"mem,tag1=val10,tag2=val20",
"mem,tag1=val11,tag2=val20",
"mem,tag1=val11,tag2=val21",
)},
group: datatypes.GroupBy,
keys: []string{"tag0", "tag2"},
exp: `group:
tag key : _m,tag0,tag1
partition key: val00,<nil>
series: _m=aaa,tag0=val00
series: _m=cpu,tag0=val00,tag1=val10
series: _m=cpu,tag0=val00,tag1=val11
series: _m=cpu,tag0=val00,tag1=val12
group:
tag key : _m,tag0
partition key: val01,<nil>
series: _m=aaa,tag0=val01
group:
tag key : _m,tag1,tag2
partition key: <nil>,val20
series: _m=mem,tag1=val10,tag2=val20
series: _m=mem,tag1=val11,tag2=val20
group:
tag key : _m,tag1,tag2
partition key: <nil>,val21
series: _m=mem,tag1=val11,tag2=val21
`,
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
newCursor := func() (reads.SeriesCursor, error) {
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
return tt.cur, nil
}
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
var hints datatypes.HintFlags
hints.SetHintSchemaAllTime()
rs := reads.NewGroupResultSet(context.Background(), &datatypes.ReadRequest{Group: tt.group, GroupKeys: tt.keys, Hints: hints}, newCursor)
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
sb := new(strings.Builder)
GroupResultSetToString(sb, rs, SkipNilCursor())
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
if got := sb.String(); !cmp.Equal(got, tt.exp) {
t.Errorf("unexpected value; -got/+exp\n%s", cmp.Diff(strings.Split(got, "\n"), strings.Split(tt.exp, "\n")))
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
}
})
}
}
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
func TestNewGroupResultSet_Sorting(t *testing.T) {
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
tests := []struct {
name string
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
keys []string
opts []reads.GroupOption
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
exp string
}{
{
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
name: "nil hi",
keys: []string{"tag0", "tag2"},
exp: `group:
tag key : _m,tag0,tag1
partition key: val00,<nil>
series: _m=aaa,tag0=val00
series: _m=cpu,tag0=val00,tag1=val10
series: _m=cpu,tag0=val00,tag1=val11
series: _m=cpu,tag0=val00,tag1=val12
group:
tag key : _m,tag0
partition key: val01,<nil>
series: _m=aaa,tag0=val01
group:
tag key : _m,tag1,tag2
partition key: <nil>,val20
series: _m=mem,tag1=val10,tag2=val20
series: _m=mem,tag1=val11,tag2=val20
group:
tag key : _m,tag1,tag2
partition key: <nil>,val21
series: _m=mem,tag1=val11,tag2=val21
`,
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
},
{
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
name: "nil lo",
keys: []string{"tag0", "tag2"},
opts: []reads.GroupOption{reads.GroupOptionNilSortLo()},
exp: `group:
tag key : _m,tag1,tag2
partition key: <nil>,val20
series: _m=mem,tag1=val11,tag2=val20
series: _m=mem,tag1=val10,tag2=val20
group:
tag key : _m,tag1,tag2
partition key: <nil>,val21
series: _m=mem,tag1=val11,tag2=val21
group:
tag key : _m,tag0,tag1
partition key: val00,<nil>
series: _m=cpu,tag0=val00,tag1=val10
series: _m=cpu,tag0=val00,tag1=val11
series: _m=cpu,tag0=val00,tag1=val12
series: _m=aaa,tag0=val00
group:
tag key : _m,tag0
partition key: val01,<nil>
series: _m=aaa,tag0=val01
`,
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
newCursor := func() (reads.SeriesCursor, error) {
return &sliceSeriesCursor{
rows: newSeriesRows(
"aaa,tag0=val00",
"aaa,tag0=val01",
"cpu,tag0=val00,tag1=val10",
"cpu,tag0=val00,tag1=val11",
"cpu,tag0=val00,tag1=val12",
"mem,tag1=val10,tag2=val20",
"mem,tag1=val11,tag2=val20",
"mem,tag1=val11,tag2=val21",
)}, nil
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
}
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
var hints datatypes.HintFlags
hints.SetHintSchemaAllTime()
rs := reads.NewGroupResultSet(context.Background(), &datatypes.ReadRequest{Group: datatypes.GroupBy, GroupKeys: tt.keys, Hints: hints}, newCursor, tt.opts...)
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
sb := new(strings.Builder)
GroupResultSetToString(sb, rs, SkipNilCursor())
if got := sb.String(); !cmp.Equal(got, tt.exp) {
t.Errorf("unexpected value; -got/+exp\n%s", cmp.Diff(strings.Split(got, "\n"), strings.Split(tt.exp, "\n")))
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
}
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
})
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
}
}
type sliceSeriesCursor struct {
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
rows []reads.SeriesRow
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
i int
}
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
func newSeriesRows(keys ...string) []reads.SeriesRow {
rows := make([]reads.SeriesRow, len(keys))
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
for i := range keys {
rows[i].Name, rows[i].SeriesTags = models.ParseKeyBytes([]byte(keys[i]))
rows[i].Tags = rows[i].SeriesTags.Clone()
rows[i].Tags.Set([]byte("_m"), rows[i].Name)
}
return rows
}
func (s *sliceSeriesCursor) Close() {}
func (s *sliceSeriesCursor) Err() error { return nil }
chore(reads): Document ResultSet and GroupResultSet APIs; add func Err * add option to Group sort to specify precedence of `nil` * improve unit tests for GroupResultSet implementation
2018-11-20 18:38:36 +00:00
func (s *sliceSeriesCursor) Next() *reads.SeriesRow {
chore: refactor reads service and make it consumable externally This pulls in the code that allows doing reads with flux into the platform repo, and removes extra.go. The reusable portion is under storage/reads, where the concrete implementation for one of the platform's engines is in storage/readservice. In order to make this more reusable, the cursors had to move into their own package, decoupling it from all of the other code in the tsdb package. tsdb/cursors is this new package, and type/function aliases have been added to the tsdb package to point at it. The models package already is very light on transitive dependencies and so it was allowed to be depended on in a concrete way in the cursors package. Finally, the protobuf definitions for issuing GRPC reads has been moved into its own package for two reasons: 1. It's a clean separation, and helps keep it that way. 2. Many/most consumers will not be using GRPC. We just use the datatypes to express the API which helps making a GRPC server easier. It is left up to future refactorings (specifically ones that involve GPRC) to determine if these types should remain, or if there is a cleaner way. There's still some dependencies on both github.com/influxdata/influxql and github.com/influxdata/influxdb/logger that we can hopefully remove in future refactorings.
2018-10-05 22:02:31 +00:00
if s.i < len(s.rows) {
s.i++
return &s.rows[s.i-1]
}
return nil
}
chore: Add GroupBy benchmark
2018-11-20 18:55:04 +00:00
func BenchmarkNewGroupResultSet_GroupBy(b *testing.B) {
card := []int{10, 10, 10}
chore(gen): Add ingen generator data structures to platform for reuse
2018-12-10 20:53:16 +00:00
vals := make([]gen.CountableSequence, len(card))
chore: Add GroupBy benchmark
2018-11-20 18:55:04 +00:00
for i := range card {
vals[i] = gen.NewCounterByteSequenceCount(card[i])
}
tags := gen.NewTagsValuesSequenceValues("tag", vals)
rows := make([]reads.SeriesRow, tags.Count())
for i := range rows {
tags.Next()
t := tags.Value().Clone()
rows[i].SeriesTags = t
rows[i].Tags = t
rows[i].Name = []byte("m0")
}
cur := &sliceSeriesCursor{rows: rows}
newCursor := func() (reads.SeriesCursor, error) {
cur.i = 0
return cur, nil
}
b.ResetTimer()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
var hints datatypes.HintFlags
hints.SetHintSchemaAllTime()
rs := reads.NewGroupResultSet(context.Background(), &datatypes.ReadRequest{Group: datatypes.GroupBy, GroupKeys: []string{"tag2"}, Hints: hints}, newCursor)
rs.Close()
}
}