Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
influxdb/models/points_test.go

2395 lines
65 KiB
Go
Raw Normal View History

refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
package models_test
Move data.Point to tsdb.Point
2015-05-22 21:00:51 +00:00
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
import (
"bytes"
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
"fmt"
Add point.UnmarshalBinary() bounds checking.
2017-03-01 18:43:13 +00:00
"io"
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
"math"
skip empty string for start position
2015-10-28 14:12:53 +00:00
"math/rand"
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"reflect"
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
"strconv"
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
"strings"
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"testing"
"time"
refactor tsdb tests into test package
2015-07-20 19:59:46 +00:00
rename influxdb/influxdb to influxdata/influxdb This commit changes all the import and URL references from: github.com/influxdb/influxdb to: github.com/influxdata/influxdb
2016-02-10 17:26:18 +00:00
"github.com/influxdata/influxdb/models"
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
)
Move data.Point to tsdb.Point
2015-05-22 21:00:51 +00:00
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
var (
udp client: large points will now be split, if possible The v2 UDP client will attempt to split points that exceed the configured payload size. It will only do this for points that have a timestamp specified.
2016-09-14 01:15:41 +00:00
tags = models.NewTags(map[string]string{"foo": "bar", "apple": "orange", "host": "serverA", "region": "uswest"})
fields = models.Fields{
"int64": int64(math.MaxInt64),
"uint32": uint32(math.MaxUint32),
"string": "String field that has a decent length, probably some log message or something",
"boolean": false,
models: Added AppendString, PointSize, and Round to Point This change also updates the UDP client to take advantage of these improvements, as well as some code review changes.
2016-09-14 15:55:31 +00:00
"float64-tiny": float64(math.SmallestNonzeroFloat64),
"float64-large": float64(math.MaxFloat64),
udp client: large points will now be split, if possible The v2 UDP client will attempt to split points that exceed the configured payload size. It will only do this for points that have a timestamp specified.
2016-09-14 01:15:41 +00:00
}
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
maxFloat64 = strconv.FormatFloat(math.MaxFloat64, 'f', 1, 64)
minFloat64 = strconv.FormatFloat(-math.MaxFloat64, 'f', 1, 64)
Use strings.Replacer to escape string field benchmark old ns/op new ns/op delta BenchmarkEscapeStringField_Plain-4 167 65.3 -60.90% BenchmarkEscapeString_Quotes-4 167 165 -1.20% BenchmarkEscapeString_Backslashes-4 211 184 -12.80% BenchmarkEscapeString_QuotesAndBackslashes-4 413 397 -3.87% BenchmarkExportTSMStrings_100s_250vps-4 33833611 27381442 -19.07% BenchmarkExportWALStrings_100s_250vps-4 34977761 29222717 -16.45% benchmark old allocs new allocs delta BenchmarkEscapeStringField_Plain-4 4 1 -75.00% BenchmarkEscapeString_Quotes-4 4 3 -25.00% BenchmarkEscapeString_Backslashes-4 5 3 -40.00% BenchmarkEscapeString_QuotesAndBackslashes-4 9 5 -44.44% BenchmarkExportTSMStrings_100s_250vps-4 201605 76938 -61.84% BenchmarkExportWALStrings_100s_250vps-4 225371 100728 -55.31% benchmark old bytes new bytes delta BenchmarkEscapeStringField_Plain-4 56 16 -71.43% BenchmarkEscapeString_Quotes-4 56 48 -14.29% BenchmarkEscapeString_Backslashes-4 104 80 -23.08% BenchmarkEscapeString_QuotesAndBackslashes-4 208 160 -23.08% BenchmarkExportTSMStrings_100s_250vps-4 10872629 6062048 -44.24% BenchmarkExportWALStrings_100s_250vps-4 10094933 5269980 -47.80%
2016-12-18 07:37:25 +00:00
sink interface{}
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
)
Move data.Point to tsdb.Point
2015-05-22 21:00:51 +00:00
func TestMarshal(t *testing.T) {
refactor tsdb tests into test package
2015-07-20 19:59:46 +00:00
got := tags.HashKey()
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
if exp := ",apple=orange,foo=bar,host=serverA,region=uswest"; string(got) != exp {
Move data.Point to tsdb.Point
2015-05-22 21:00:51 +00:00
t.Log("got: ", string(got))
t.Log("exp: ", exp)
t.Error("invalid match")
}
}
Add support for uint64 in the clients
2017-10-04 20:53:45 +00:00
func TestMarshalFields(t *testing.T) {
for _, tt := range []struct {
name string
value interface{}
exp string
}{
{
name: "Float",
value: float64(2),
exp: `value=2`,
},
{
name: "Integer",
value: int64(2),
exp: `value=2i`,
},
{
name: "Unsigned",
value: uint64(2),
exp: `value=2u`,
},
{
name: "String",
value: "foobar",
exp: `value="foobar"`,
},
{
name: "Boolean",
value: true,
exp: `value=true`,
},
} {
t.Run(tt.name, func(t *testing.T) {
fields := map[string]interface{}{"value": tt.value}
if have, want := models.Fields(fields).MarshalBinary(), []byte(tt.exp); !bytes.Equal(have, want) {
t.Fatalf("unexpected field output: %s != %s", string(have), string(want))
}
})
}
}
Fixes issue #8199 Benchmarks ``` benchmark old ns/op new ns/op delta BenchmarkMarshal-8 1216 1007 -17.19% benchmark old allocs new allocs delta BenchmarkMarshal-8 4 2 -50.00% benchmark old bytes new bytes delta BenchmarkMarshal-8 416 256 -38.46% ```
2017-03-25 20:33:36 +00:00
func TestTags_HashKey(t *testing.T) {
tags = models.NewTags(map[string]string{"A FOO": "bar", "APPLE": "orange", "host": "serverA", "region": "uswest"})
got := tags.HashKey()
if exp := ",A\\ FOO=bar,APPLE=orange,host=serverA,region=uswest"; string(got) != exp {
t.Log("got: ", string(got))
t.Log("exp: ", exp)
t.Error("invalid match")
}
}
Move data.Point to tsdb.Point
2015-05-22 21:00:51 +00:00
func BenchmarkMarshal(b *testing.B) {
for i := 0; i < b.N; i++ {
refactor tsdb tests into test package
2015-07-20 19:59:46 +00:00
tags.HashKey()
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
models: Added AppendString, PointSize, and Round to Point This change also updates the UDP client to take advantage of these improvements, as well as some code review changes.
2016-09-14 15:55:31 +00:00
func TestPoint_StringSize(t *testing.T) {
testPoint_cube(t, func(p models.Point) {
l := p.StringSize()
s := p.String()
if l != len(s) {
t.Errorf("Incorrect length for %q. got %v, exp %v", s, l, len(s))
}
})
}
func TestPoint_AppendString(t *testing.T) {
testPoint_cube(t, func(p models.Point) {
got := p.AppendString(nil)
exp := []byte(p.String())
if !reflect.DeepEqual(exp, got) {
t.Errorf("AppendString() didn't match String(): got %v, exp %v", got, exp)
}
})
}
func testPoint_cube(t *testing.T, f func(p models.Point)) {
// heard of a table-driven test? let's make a cube-driven test...
Add Size to models.Tags
2017-02-08 18:44:48 +00:00
tagList := []models.Tags{nil, {models.NewTag([]byte("foo"), []byte("bar"))}, tags}
models: Added AppendString, PointSize, and Round to Point This change also updates the UDP client to take advantage of these improvements, as well as some code review changes.
2016-09-14 15:55:31 +00:00
fieldList := []models.Fields{{"a": 42.0}, {"a": 42, "b": "things"}, fields}
timeList := []time.Time{time.Time{}, time.Unix(0, 0), time.Unix(-34526, 0), time.Unix(231845, 0), time.Now()}
for _, tagSet := range tagList {
for _, fieldSet := range fieldList {
for _, pointTime := range timeList {
p, err := models.NewPoint("test", tagSet, fieldSet, pointTime)
if err != nil {
t.Errorf("unexpected error creating point: %v", err)
continue
}
f(p)
}
}
}
}
Fix memory leak of retained HTTP write payloads This leak seems to have been introduced in 8aa224b22dc007e, present in 1.1.0 and 1.1.1. When points were parsed from HTTP payloads, their tags and fields referred to subslices of the request body; if any tag set introduced a new series, then those tags then were stored in the in-memory series index objects, preventing the HTTP body from being garbage collected. If there were no new series in the payload, then the request body would be garbage collected as usual. Now, we clone the tags before we store them in the index. This is an imperfect fix because the Point still holds references to the original tags, and the Point's field iterator also refers to the payload buffer. However, the current write code path does not retain references to the Point or its fields; and this change will likely be obsoleted when TSI is introduced. This change likely fixes #7827, #7810, #7778, and perhaps others.
2017-01-13 00:16:54 +00:00
func TestTag_Clone(t *testing.T) {
Add Size to models.Tags
2017-02-08 18:44:48 +00:00
tag := models.NewTag([]byte("key"), []byte("value"))
Fix memory leak of retained HTTP write payloads This leak seems to have been introduced in 8aa224b22dc007e, present in 1.1.0 and 1.1.1. When points were parsed from HTTP payloads, their tags and fields referred to subslices of the request body; if any tag set introduced a new series, then those tags then were stored in the in-memory series index objects, preventing the HTTP body from being garbage collected. If there were no new series in the payload, then the request body would be garbage collected as usual. Now, we clone the tags before we store them in the index. This is an imperfect fix because the Point still holds references to the original tags, and the Point's field iterator also refers to the payload buffer. However, the current write code path does not retain references to the Point or its fields; and this change will likely be obsoleted when TSI is introduced. This change likely fixes #7827, #7810, #7778, and perhaps others.
2017-01-13 00:16:54 +00:00
c := tag.Clone()
if &c.Key == &tag.Key || !bytes.Equal(c.Key, tag.Key) {
t.Fatalf("key %s should have been a clone of %s", c.Key, tag.Key)
}
if &c.Value == &tag.Value || !bytes.Equal(c.Value, tag.Value) {
t.Fatalf("value %s should have been a clone of %s", c.Value, tag.Value)
}
}
func TestTags_Clone(t *testing.T) {
tags := models.NewTags(map[string]string{"k1": "v1", "k2": "v2", "k3": "v3"})
clone := tags.Clone()
for i := range tags {
tag := tags[i]
c := clone[i]
if &c.Key == &tag.Key || !bytes.Equal(c.Key, tag.Key) {
t.Fatalf("key %s should have been a clone of %s", c.Key, tag.Key)
}
if &c.Value == &tag.Value || !bytes.Equal(c.Value, tag.Value) {
t.Fatalf("value %s should have been a clone of %s", c.Value, tag.Value)
}
}
}
udp client: large points will now be split, if possible The v2 UDP client will attempt to split points that exceed the configured payload size. It will only do this for points that have a timestamp specified.
2016-09-14 01:15:41 +00:00
var p models.Point
func BenchmarkNewPoint(b *testing.B) {
ts := time.Now()
for i := 0; i < b.N; i++ {
p, _ = models.NewPoint("measurement", tags, fields, ts)
}
}
Optimize Point.NewPointFromBytes There was a check to ensure that fields exists when unmarshalBinary is called. This created a map and other garbage just to see if any fields exist. This changes it to use a FieldIterator that does not allocate as much as the other method.
2017-04-06 18:46:09 +00:00
func BenchmarkNewPointFromBinary(b *testing.B) {
pts, err := models.ParsePointsString("cpu value1=1.0,value2=1.0,value3=3.0,value4=4,value5=\"five\" 1000000000")
if err != nil {
b.Fatalf("unexpected error ParsePointsString: %v", err)
}
bytes, err := pts[0].MarshalBinary()
if err != nil {
b.Fatalf("unexpected error MarshalBinary: %v", err)
}
for i := 0; i < b.N; i++ {
_, err := models.NewPointFromBytes(bytes)
if err != nil {
b.Fatalf("unexpected error NewPointsFromBytes: %v", err)
}
}
}
Pre-allocated Points slice when parsing points Over a longer period of writes, this allocation shows up quite a bit in profiles since the slice needs to be resized frequently. This scans the slice to count how many lines are going to be parsed in order to pre-allocate the slice capacity. It's slightly slower, but creates less garbage in the long run.
2016-09-20 22:27:19 +00:00
func BenchmarkParsePointNoTags5000(b *testing.B) {
var batch [5000]string
for i := 0; i < len(batch); i++ {
batch[i] = `cpu value=1i 1000000000`
}
lines := strings.Join(batch[:], "\n")
b.ResetTimer()
for i := 0; i < b.N; i++ {
models.ParsePoints([]byte(lines))
b.SetBytes(int64(len(lines)))
}
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
func BenchmarkParsePointNoTags(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
}
}
add bounds checking for shard group timestamps enforce strict valid time ranges on points small
2015-12-07 18:01:46 +00:00
func BenchmarkParsePointWithPrecisionN(b *testing.B) {
line := `cpu value=1i 1000000000`
defaultTime := time.Now().UTC()
for i := 0; i < b.N; i++ {
models.ParsePointsWithPrecision([]byte(line), defaultTime, "n")
b.SetBytes(int64(len(line)))
}
}
func BenchmarkParsePointWithPrecisionU(b *testing.B) {
line := `cpu value=1i 1000000000`
defaultTime := time.Now().UTC()
for i := 0; i < b.N; i++ {
models.ParsePointsWithPrecision([]byte(line), defaultTime, "u")
b.SetBytes(int64(len(line)))
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func BenchmarkParsePointsTagsSorted2(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu,host=serverA,region=us-west value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func BenchmarkParsePointsTagsSorted5(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu,env=prod,host=serverA,region=us-west,target=servers,zone=1c value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func BenchmarkParsePointsTagsSorted10(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu,env=prod,host=serverA,region=us-west,tag1=value1,tag2=value2,tag3=value3,tag4=value4,tag5=value5,target=servers,zone=1c value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func BenchmarkParsePointsTagsUnSorted2(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu,region=us-west,host=serverA value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pt, _ := models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
pt[0].Key()
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func BenchmarkParsePointsTagsUnSorted5(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu,region=us-west,host=serverA,env=prod,target=servers,zone=1c value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pt, _ := models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
pt[0].Key()
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func BenchmarkParsePointsTagsUnSorted10(b *testing.B) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
line := `cpu,region=us-west,host=serverA,env=prod,target=servers,zone=1c,tag1=value1,tag2=value2,tag3=value3,tag4=value4,tag5=value5 value=1i 1000000000`
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
for i := 0; i < b.N; i++ {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pt, _ := models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
b.SetBytes(int64(len(line)))
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
pt[0].Key()
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
Add ParseKey benchmark
2016-05-20 20:24:36 +00:00
func BenchmarkParseKey(b *testing.B) {
line := `cpu,region=us-west,host=serverA,env=prod,target=servers,zone=1c,tag1=value1,tag2=value2,tag3=value3,tag4=value4,tag5=value5`
for i := 0; i < b.N; i++ {
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.ParseKey([]byte(line))
Add ParseKey benchmark
2016-05-20 20:24:36 +00:00
}
}
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
// TestPoint wraps a models.Point but also makes available the raw
// arguments to the Point.
//
// This is useful for ensuring that comparisons between results of
// operations on Points match the expected input data to the Point,
// since models.Point does not expose the raw input data (e.g., tags)
// via its API.
type TestPoint struct {
RawFields models.Fields
RawTags models.Tags
RawTime time.Time
models.Point
}
// NewTestPoint returns a new TestPoint.
//
// NewTestPoint panics if it is not a valid models.Point.
func NewTestPoint(name string, tags models.Tags, fields models.Fields, time time.Time) TestPoint {
return TestPoint{
RawTags: tags,
RawFields: fields,
RawTime: time,
Point: models.MustNewPoint(name, tags, fields, time),
}
}
func test(t *testing.T, line string, point TestPoint) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsWithPrecision([]byte(line), time.Unix(0, 0), "n")
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
if err != nil {
t.Fatalf(`ParsePoints("%s") mismatch. got %v, exp nil`, line, err)
}
if exp := 1; len(pts) != exp {
t.Fatalf(`ParsePoints("%s") len mismatch. got %d, exp %d`, line, len(pts), exp)
}
if exp := point.Key(); !bytes.Equal(pts[0].Key(), exp) {
t.Errorf("ParsePoints(\"%s\") key mismatch.\ngot %v\nexp %v", line, string(pts[0].Key()), string(exp))
}
if exp := len(point.Tags()); len(pts[0].Tags()) != exp {
t.Errorf(`ParsePoints("%s") tags mismatch. got %v, exp %v`, line, pts[0].Tags(), exp)
}
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
for _, tag := range pts[0].Tags() {
if !bytes.Equal(tag.Value, point.RawTags.Get(tag.Key)) {
t.Errorf(`ParsePoints("%s") tags mismatch. got %s, exp %s`, line, tag.Value, point.RawTags.Get(tag.Key))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
for name, value := range point.RawFields {
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := pts[0].Fields()
if err != nil {
t.Fatal(err)
}
val := fields[name]
Parse NaN as float Fixes #3230
2015-07-06 16:18:07 +00:00
expfval, ok := val.(float64)
if ok && math.IsNaN(expfval) {
gotfval, ok := value.(float64)
if ok && !math.IsNaN(gotfval) {
t.Errorf(`ParsePoints("%s") field '%s' mismatch. exp NaN`, line, name)
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
}
if !reflect.DeepEqual(val, value) {
t.Errorf(`ParsePoints("%s") field '%s' mismatch. got %[3]v (%[3]T), exp %[4]v (%[4]T)`, line, name, val, value)
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
if !pts[0].Time().Equal(point.Time()) {
t.Errorf(`ParsePoints("%s") time mismatch. got %v, exp %v`, line, pts[0].Time(), point.Time())
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
if !strings.HasPrefix(pts[0].String(), line) {
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
t.Errorf("ParsePoints string mismatch.\ngot: %v\nexp: %v", pts[0].String(), line)
}
}
func TestParsePointNoValue(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsString("")
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, "", err)
}
if exp := 0; len(pts) != exp {
Fix panics found via go-fuzz Fixes #3288
2015-08-05 22:41:59 +00:00
t.Errorf(`ParsePoints("%s") len mismatch. got %v, exp %v`, "", len(pts), exp)
}
}
func TestParsePointWhitespaceValue(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsString(" ")
Fix panics found via go-fuzz Fixes #3288
2015-08-05 22:41:59 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, "", err)
}
if exp := 0; len(pts) != exp {
t.Errorf(`ParsePoints("%s") len mismatch. got %v, exp %v`, "", len(pts), exp)
}
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
func TestParsePointNoFields(t *testing.T) {
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
expectedSuffix := "missing fields"
examples := []string{
"cpu_load_short,host=server01,region=us-west",
"cpu",
Improve scanning performance
2015-11-20 01:41:25 +00:00
"cpu,host==",
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
"=",
}
for i, example := range examples {
_, err := models.ParsePointsString(example)
if err == nil {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got nil, exp error`, i, example)
} else if !strings.HasSuffix(err.Error(), expectedSuffix) {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got %q, exp suffix %q`, i, example, err, expectedSuffix)
}
Return parsing error when tag name is missing Fixes #2678
2015-06-26 18:34:35 +00:00
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
func TestParsePointNoTimestamp(t *testing.T) {
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
test(t, "cpu value=1", NewTestPoint("cpu", nil, models.Fields{"value": 1.0}, time.Unix(0, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
func TestParsePointMissingQuote(t *testing.T) {
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
expectedSuffix := "unbalanced quotes"
examples := []string{
`cpu,host=serverA value="test`,
`cpu,host=serverA value="test""`,
}
for i, example := range examples {
_, err := models.ParsePointsString(example)
if err == nil {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got nil, exp error`, i, example)
} else if !strings.HasSuffix(err.Error(), expectedSuffix) {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got %q, exp suffix %q`, i, example, err, expectedSuffix)
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
Update points_test.go
2015-10-13 23:46:23 +00:00
func TestParsePointMissingTagKey(t *testing.T) {
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
expectedSuffix := "missing tag key"
examples := []string{
`cpu, value=1`,
`cpu,`,
Add extra test cases.
2015-11-20 11:01:23 +00:00
`cpu,,,`,
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
`cpu,host=serverA,=us-east value=1i`,
`cpu,host=serverAa\,,=us-east value=1i`,
`cpu,host=serverA\,,=us-east value=1i`,
`cpu, =serverA value=1i`,
}
for i, example := range examples {
_, err := models.ParsePointsString(example)
if err == nil {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got nil, exp error`, i, example)
} else if !strings.HasSuffix(err.Error(), expectedSuffix) {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got %q, exp suffix %q`, i, example, err, expectedSuffix)
}
Return parsing error when tag name is missing Fixes #2678
2015-06-26 18:34:35 +00:00
}
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,\ =us-east value=1i`)
Return parsing error when tag name is missing Fixes #2678
2015-06-26 18:34:35 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,\ =us-east value=1i`, err)
Return parsing error when tag name is missing Fixes #2678
2015-06-26 18:34:35 +00:00
}
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
func TestParsePointMissingTagValue(t *testing.T) {
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
expectedSuffix := "missing tag value"
examples := []string{
Add extra test coverage; fix test typo
2015-11-18 23:18:28 +00:00
`cpu,host`,
`cpu,host,`,
Fix a panic when a tag value was empty A panic would happen if you wrote the following: cpu,host= There was a missing bounds check when scanning the tag value. Fixes #5262.
2016-01-15 22:07:34 +00:00
`cpu,host=`,
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
`cpu,host value=1i`,
`cpu,host=serverA,region value=1i`,
`cpu,host=serverA,region= value=1i`,
`cpu,host=serverA,region=,zone=us-west value=1i`,
}
for i, example := range examples {
_, err := models.ParsePointsString(example)
if err == nil {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got nil, exp error`, i, example)
} else if !strings.HasSuffix(err.Error(), expectedSuffix) {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got %q, exp suffix %q`, i, example, err, expectedSuffix)
}
Fix line protocol accepting tags with no values If a tag with no value was in the middle of all the tags, it would get accepted as valid incorrectly. Fixes #4421
2015-10-13 15:43:13 +00:00
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
Add extra test coverage; fix test typo
2015-11-18 23:18:28 +00:00
func TestParsePointInvalidTagFormat(t *testing.T) {
expectedSuffix := "invalid tag format"
examples := []string{
`cpu,host=f=o,`,
Add extra test cases.
2015-11-20 11:01:23 +00:00
`cpu,host=f\==o,`,
Add extra test coverage; fix test typo
2015-11-18 23:18:28 +00:00
}
for i, example := range examples {
_, err := models.ParsePointsString(example)
if err == nil {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got nil, exp error`, i, example)
} else if !strings.HasSuffix(err.Error(), expectedSuffix) {
t.Errorf(`[Example %d] ParsePoints("%s") mismatch. got %q, exp suffix %q`, i, example, err, expectedSuffix)
}
}
}
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
func TestParsePointMissingFieldName(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west =`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west =`)
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west =123i`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west =123i`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west a\ =123i`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west a\ =123i`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=123i,=456i`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=123i,=456i`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
}
}
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
func TestParsePointMissingFieldValue(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=`)
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
if err == nil {
Fix invalid field value format not parsed correctly A field value of just a numeric value would be accepted by the line protocol parser but the value would be set as the field name and the value would be nil. Instead, return an error because all field values need a field name.
2015-06-22 16:23:37 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=`)
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value= 1000000000i`)
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value= 1000000000i`)
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=,value2=1i`)
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=,value2=1i`)
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
}
Fix invalid field value format not parsed correctly A field value of just a numeric value would be accepted by the line protocol parser but the value would be set as the field name and the value would be nil. Instead, return an error because all field values need a field name.
2015-06-22 16:23:37 +00:00
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=server01,region=us-west 1434055562000000000i`)
Fix invalid field value format not parsed correctly A field value of just a numeric value would be accepted by the line protocol parser but the value would be set as the field name and the value would be nil. Instead, return an error because all field values need a field name.
2015-06-22 16:23:37 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=server01,region=us-west 1434055562000000000i`)
Fix invalid field value format not parsed correctly A field value of just a numeric value would be accepted by the line protocol parser but the value would be set as the field name and the value would be nil. Instead, return an error because all field values need a field name.
2015-06-22 16:23:37 +00:00
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=server01,region=us-west value=1i,b`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=server01,region=us-west value=1i,b`)
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
}
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
}
func TestParsePointBadNumber(t *testing.T) {
Update number scanning edge cases This should fix #5965, and other issues that result from submitting malformed numbers with points
2016-03-14 20:45:15 +00:00
for _, tt := range []string{
"cpu v=- ",
"cpu v=-i ",
"cpu v=-. ",
"cpu v=. ",
"cpu v=1.0i ",
"cpu v=1ii ",
"cpu v=1a ",
"cpu v=-e-e-e ",
"cpu v=42+3 ",
"cpu v= ",
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
"cpu v=-123u",
Update number scanning edge cases This should fix #5965, and other issues that result from submitting malformed numbers with points
2016-03-14 20:45:15 +00:00
} {
_, err := models.ParsePointsString(tt)
if err == nil {
t.Errorf("Point %q should be invalid", tt)
}
add point conversion test for 1.0i
2015-07-31 01:55:08 +00:00
}
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
}
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
func TestParsePointMaxInt64(t *testing.T) {
// out of range
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=9223372036854775808i`)
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
exp := `unable to parse 'cpu,host=serverA,region=us-west value=9223372036854775808i': unable to parse integer 9223372036854775808: strconv.ParseInt: parsing "9223372036854775808": value out of range`
if err == nil || (err != nil && err.Error() != exp) {
t.Fatalf("Error mismatch:\nexp: %s\ngot: %v", exp, err)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
}
// max int
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
p, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=9223372036854775807i`)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Fatalf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=9223372036854775807i`, err)
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := p[0].Fields()
if err != nil {
t.Fatal(err)
}
if exp, got := int64(9223372036854775807), fields["value"].(int64); exp != got {
Fix typos. * meaining --> meaning * communcation --> communication * deterimine --> determine * mistmatch --> mistmatch
2015-10-17 14:50:45 +00:00
t.Fatalf("ParsePoints Value mismatch. \nexp: %v\ngot: %v", exp, got)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
}
// leading zeros
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=0009223372036854775807i`)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Fatalf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=0009223372036854775807i`, err)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
}
}
func TestParsePointMinInt64(t *testing.T) {
// out of range
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=-9223372036854775809i`)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err == nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=-9223372036854775809i`)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
}
// min int
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
p, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=-9223372036854775808i`)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=-9223372036854775808i`, err)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
}
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
fields, err := p[0].Fields()
if err != nil {
t.Fatal(err)
}
if exp, got := int64(-9223372036854775808), fields["value"].(int64); exp != got {
t.Fatalf("ParsePoints Value mismatch. \nexp: %v\ngot: %v", exp, got)
}
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
// leading zeros
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=-0009223372036854775808i`)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=-0009223372036854775808i`, err)
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
}
}
func TestParsePointMaxFloat64(t *testing.T) {
// out of range
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(fmt.Sprintf(`cpu,host=serverA,region=us-west value=%s`, "1"+string(maxFloat64)))
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=...`)
}
// max float
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
p, err := models.ParsePointsString(fmt.Sprintf(`cpu,host=serverA,region=us-west value=%s`, string(maxFloat64)))
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=9223372036854775807`, err)
}
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
fields, err := p[0].Fields()
if err != nil {
t.Fatal(err)
}
if exp, got := math.MaxFloat64, fields["value"].(float64); exp != got {
t.Fatalf("ParsePoints Value mismatch. \nexp: %v\ngot: %v", exp, got)
}
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
// leading zeros
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(fmt.Sprintf(`cpu,host=serverA,region=us-west value=%s`, "0000"+string(maxFloat64)))
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=0009223372036854775807`, err)
}
}
func TestParsePointMinFloat64(t *testing.T) {
// out of range
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(fmt.Sprintf(`cpu,host=serverA,region=us-west value=%s`, "-1"+string(minFloat64)[1:]))
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=...`)
}
// min float
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
p, err := models.ParsePointsString(fmt.Sprintf(`cpu,host=serverA,region=us-west value=%s`, string(minFloat64)))
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=...`, err)
}
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
fields, err := p[0].Fields()
if err != nil {
t.Fatal(err)
}
if exp, got := -math.MaxFloat64, fields["value"].(float64); exp != got {
t.Fatalf("ParsePoints Value mismatch. \nexp: %v\ngot: %v", exp, got)
}
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
// leading zeros
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(fmt.Sprintf(`cpu,host=serverA,region=us-west value=%s`, "-0000000"+string(minFloat64)[1:]))
Prevent out of range ints and floats from being parsed sucessfully Field values that were out of range for the type would panic the database when being inserted because the parser would allow them as valid points. This change prevents those invalid values from being parsed and instead returns an error. An alternative fix considered was to handle the error and clamp the value to the min/max value for the type. This would treat numeric range errors slightly differently than other type erros which might lead to confusion. The simplest fix with the current parser would be to just convert each field to the type at parse time. Unfortunately, this adds extra memory allocations and lowers throughput significantly. Since out of range values are less common than in-range values, some heuristics are used to determine when the more expensive type parsing and range checking is performed. Essentially, we only do the slow path when we cannot determine that the value is in an acceptable type range. Fixes #3127
2015-06-25 19:20:20 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=...`, err)
}
}
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
func TestParsePointMaxUint64(t *testing.T) {
// out of range
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=18446744073709551616u`)
exp := `unable to parse 'cpu,host=serverA,region=us-west value=18446744073709551616u': unable to parse unsigned 18446744073709551616: strconv.ParseUint: parsing "18446744073709551616": value out of range`
if err == nil || (err != nil && err.Error() != exp) {
t.Fatalf("Error mismatch:\nexp: %s\ngot: %v", exp, err)
}
// max int
p, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=18446744073709551615u`)
if err != nil {
t.Fatalf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=18446744073709551615u`, err)
}
fields, err := p[0].Fields()
if err != nil {
t.Fatal(err)
}
if exp, got := uint64(18446744073709551615), fields["value"].(uint64); exp != got {
t.Fatalf("ParsePoints Value mismatch. \nexp: %v\ngot: %v", exp, got)
}
// leading zeros
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=00018446744073709551615u`)
if err != nil {
t.Fatalf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=00018446744073709551615u`, err)
}
}
func TestParsePointMinUint64(t *testing.T) {
// out of range
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=--1u`)
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=-1u`)
}
// min int
p, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=0u`)
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=0u`, err)
}
fields, err := p[0].Fields()
if err != nil {
t.Fatal(err)
}
if exp, got := uint64(0), fields["value"].(uint64); exp != got {
t.Fatalf("ParsePoints Value mismatch. \nexp: %v\ngot: %v", exp, got)
}
// leading zeros
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=0000u`)
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=0000u`, err)
}
}
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
func TestParsePointNumberNonNumeric(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=.1a`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=.1a`)
}
}
func TestParsePointNegativeWrongPlace(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=0.-1`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=0.-1`)
}
}
Reject line protocol that terminates with '-' Fixes issue #4272.
2015-10-02 03:45:10 +00:00
func TestParsePointOnlyNegativeSign(t *testing.T) {
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=-`)
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=-`)
}
}
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
func TestParsePointFloatMultipleDecimals(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1.1.1`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=1.1.1`)
}
}
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
func TestParsePointInteger(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1i`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1i`, err)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
}
}
func TestParsePointNegativeInteger(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=-1i`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=-1i`, err)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
}
}
func TestParsePointNegativeFloat(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=-1.0`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=-1.0`, err)
}
}
func TestParsePointFloatNoLeadingDigit(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=.1`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=-1.0`, err)
}
}
func TestParsePointFloatScientific(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1.0e4`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1.0e4`, err)
}
Fix panic parsing floats without decimal Fixes #3289
2015-07-10 20:18:51 +00:00
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1e4`)
Fix panic parsing floats without decimal Fixes #3289
2015-07-10 20:18:51 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1.0e4`, err)
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := pts[0].Fields()
if err != nil {
t.Fatal(err)
}
if fields["value"] != 1e4 {
Fix panic parsing floats without decimal Fixes #3289
2015-07-10 20:18:51 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1e4`, err)
}
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
}
Allow uppercase 'e' with scientific notation
2015-10-13 20:28:57 +00:00
func TestParsePointFloatScientificUpper(t *testing.T) {
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1.0E4`)
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1.0E4`, err)
}
pts, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1E4`)
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1.0E4`, err)
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := pts[0].Fields()
if err != nil {
t.Fatal(err)
}
if fields["value"] != 1e4 {
Allow uppercase 'e' with scientific notation
2015-10-13 20:28:57 +00:00
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1E4`, err)
}
}
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
func TestParsePointFloatScientificDecimal(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=1.0e-4`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=1.0e-4`, err)
}
}
func TestParsePointFloatNegativeScientific(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=-1.0e-4`)
Prevent invalid numbers from being parsed via line protocol Adds more tests for invalid numbers such as 0.1a, -2.-4, as well test for supported formats for negative and positive integers/floats as well as scientific notation.
2015-06-11 19:06:09 +00:00
if err != nil {
t.Errorf(`ParsePoints("%s") mismatch. got %v, exp nil`, `cpu,host=serverA,region=us-west value=-1.0e-4`, err)
}
Fix panic when no value is passed to a field Fixes #2927
2015-06-11 17:25:25 +00:00
}
Be more explicit about boolean value parsing Supported boolean values are now t, T, true, TRUE, f, F, false, and FALSE. This is what the strconv.ParseBool function supports with the exception of 1 and 0. 1 and 0 would be parsed as ints in the line protocol. Previously, any non-true value would be parsed as false. e.g. value=blah would parse to false. This will now return an error as parsing time.
2015-06-11 20:53:53 +00:00
func TestParsePointBooleanInvalid(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=a`)
Be more explicit about boolean value parsing Supported boolean values are now t, T, true, TRUE, f, F, false, and FALSE. This is what the strconv.ParseBool function supports with the exception of 1 and 0. 1 and 0 would be parsed as ints in the line protocol. Previously, any non-true value would be parsed as false. e.g. value=blah would parse to false. This will now return an error as parsing time.
2015-06-11 20:53:53 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=a`)
}
}
Fix panic when parsing value in scientific notation with trailing i Fixes #3583
2015-08-10 19:46:53 +00:00
func TestParsePointScientificIntInvalid(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err := models.ParsePointsString(`cpu,host=serverA,region=us-west value=9ie10`)
Fix panic when parsing value in scientific notation with trailing i Fixes #3583
2015-08-10 19:46:53 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=9ie10`)
}
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
_, err = models.ParsePointsString(`cpu,host=serverA,region=us-west value=9e10i`)
Fix panic when parsing value in scientific notation with trailing i Fixes #3583
2015-08-10 19:46:53 +00:00
if err == nil {
t.Errorf(`ParsePoints("%s") mismatch. got nil, exp error`, `cpu,host=serverA,region=us-west value=9e10i`)
}
}
Ensure points with trailing whitespace are accepted
2016-05-19 15:58:18 +00:00
func TestParsePointWhitespace(t *testing.T) {
examples := []string{
`cpu value=1.0 1257894000000000000`,
`cpu value=1.0 1257894000000000000`,
`cpu value=1.0 1257894000000000000`,
`cpu value=1.0 1257894000000000000 `,
`cpu value=1.0 1257894000000000000
`,
`cpu value=1.0 1257894000000000000
`,
}
expPoint := NewTestPoint("cpu", models.Tags{}, models.Fields{"value": 1.0}, time.Unix(0, 1257894000000000000))
for i, example := range examples {
pts, err := models.ParsePoints([]byte(example))
if err != nil {
t.Fatalf(`[Example %d] ParsePoints("%s") error. got %v, exp nil`, i, example, err)
}
if got, exp := len(pts), 1; got != exp {
t.Fatalf("[Example %d] got %d points, expected %d", i, got, exp)
}
Convert usage of strings to []byte Measurement name and field were converted between []byte and string repetively causing lots of garbage. This switches the code to use []byte in the write path.
2017-05-09 05:20:29 +00:00
if got, exp := string(pts[0].Name()), string(expPoint.Name()); got != exp {
Fix vet issue
2016-05-27 09:31:20 +00:00
t.Fatalf("[Example %d] got %v measurement, expected %v", i, got, exp)
Ensure points with trailing whitespace are accepted
2016-05-19 15:58:18 +00:00
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := pts[0].Fields()
if err != nil {
t.Fatal(err)
}
eFields, err := expPoint.Fields()
if err != nil {
t.Fatal(err)
}
if got, exp := len(fields), len(eFields); got != exp {
Ensure points with trailing whitespace are accepted
2016-05-19 15:58:18 +00:00
t.Fatalf("[Example %d] got %d fields, expected %d", i, got, exp)
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
if got, exp := fields["value"], eFields["value"]; got != exp {
Ensure points with trailing whitespace are accepted
2016-05-19 15:58:18 +00:00
t.Fatalf(`[Example %d] got %v for field "value", expected %v`, i, got, exp)
}
if got, exp := pts[0].Time().UnixNano(), expPoint.Time().UnixNano(); got != exp {
t.Fatalf(`[Example %d] got %d time, expected %d`, i, got, exp)
}
}
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
func TestParsePointUnescape(t *testing.T) {
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
// commas in measurement name
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
test(t, `foo\,bar value=1i`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Handle escaped commas in measurement name Fixes #3183
2015-06-29 21:15:50 +00:00
"foo,bar", // comma in the name
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"value": int64(1),
Handle escaped commas in measurement name Fixes #3183
2015-06-29 21:15:50 +00:00
},
time.Unix(0, 0)))
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
// comma in measurement name with tags
test(t, `cpu\,main,regions=east value=1.0`,
NewTestPoint(
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"cpu,main", // comma in the name
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"regions": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": 1.0,
},
time.Unix(0, 0)))
Add extra test cases.
2015-11-20 11:01:23 +00:00
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
// spaces in measurement name
test(t, `cpu\ load,region=east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"cpu load", // space in the name
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"region": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": 1.0,
},
time.Unix(0, 0)))
Add extra test cases.
2015-11-20 11:01:23 +00:00
// equals in measurement name
test(t, `cpu\=load,region=east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Add extra test cases.
2015-11-20 11:01:23 +00:00
`cpu\=load`, // backslash is literal
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add extra test cases.
2015-11-20 11:01:23 +00:00
"region": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Add extra test cases.
2015-11-20 11:01:23 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
// equals in measurement name
test(t, `cpu=load,region=east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Add extra test cases.
2015-11-20 11:01:23 +00:00
`cpu=load`, // literal equals is fine in measurement name
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add extra test cases.
2015-11-20 11:01:23 +00:00
"region": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Add extra test cases.
2015-11-20 11:01:23 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
// commas in tag names
test(t, `cpu,region\,zone=east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Update points_test.go
2015-10-13 23:46:23 +00:00
"region,zone": "east", // comma in the tag key
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": 1.0,
},
time.Unix(0, 0)))
Add extra test cases.
2015-11-20 11:01:23 +00:00
// spaces in tag name
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
test(t, `cpu,region\ zone=east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add extra test cases.
2015-11-20 11:01:23 +00:00
"region zone": "east", // space in the tag name
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": 1.0,
},
time.Unix(0, 0)))
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
// backslash with escaped equals in tag name
test(t, `cpu,reg\\=ion=east value=1.0`,
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
`reg\=ion`: "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
// space is tag name
test(t, `cpu,\ =east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add extra test cases.
2015-11-20 11:01:23 +00:00
" ": "east", // tag name is single space
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
// commas in tag values
test(t, `cpu,regions=east\,west value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"regions": "east,west", // comma in the tag value
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"value": 1.0,
},
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
time.Unix(0, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
// backslash literal followed by escaped space
test(t, `cpu,regions=\\ east value=1.0`,
NewTestPoint(
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"regions": `\ east`,
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
// backslash literal followed by escaped space
test(t, `cpu,regions=eas\\ t value=1.0`,
NewTestPoint(
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"regions": `eas\ t`,
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
// backslash literal followed by trailing space
test(t, `cpu,regions=east\\ value=1.0`,
NewTestPoint(
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"regions": `east\ `,
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
// spaces in tag values
test(t, `cpu,regions=east\ west value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"regions": "east west", // comma in the tag value
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"value": 1.0,
},
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
time.Unix(0, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
Update points_test.go
2015-10-13 23:53:50 +00:00
// commas in field keys
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
test(t, `cpu,regions=east value\,ms=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"regions": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Update points_test.go
2015-10-13 23:53:50 +00:00
"value,ms": 1.0, // comma in the field keys
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
},
time.Unix(0, 0)))
Update points_test.go
2015-10-13 23:53:50 +00:00
// spaces in field keys
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
test(t, `cpu,regions=east value\ ms=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"regions": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Update points_test.go
2015-10-13 23:53:50 +00:00
"value ms": 1.0, // comma in the field keys
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
},
time.Unix(0, 0)))
do not include empty tags in hash
2015-10-04 18:40:14 +00:00
// tag with no value
test(t, `cpu,regions=east value="1"`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
do not include empty tags in hash
2015-10-04 18:40:14 +00:00
"regions": "east",
"foobar": "",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
do not include empty tags in hash
2015-10-04 18:40:14 +00:00
models.Fields{
"value": "1",
},
time.Unix(0, 0)))
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
// commas in field values
test(t, `cpu,regions=east value="1,0"`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"regions": "east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": "1,0", // comma in the field value
},
time.Unix(0, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
// random character escaped
test(t, `cpu,regions=eas\t value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"regions": "eas\\t",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"value": 1.0,
},
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
time.Unix(0, 0)))
Fix parse stuck in loop causing 500 timeout Fixes #3411
2015-07-21 17:12:40 +00:00
Add extra test cases.
2015-11-20 11:01:23 +00:00
// backslash literal followed by escaped characters
test(t, `cpu,regions=\\,\,\=east value=1.0`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add extra test cases.
2015-11-20 11:01:23 +00:00
"regions": `\,,=east`,
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
Improve robustness of tests, highlighting #3070 and #4770
2015-11-12 15:25:33 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, 0)))
Update points_test.go
2015-10-13 23:53:50 +00:00
// field keys using escape char.
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
test(t, `cpu \a=1i`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix parse stuck in loop causing 500 timeout Fixes #3411
2015-07-21 17:12:40 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"\\a": int64(1), // Left as parsed since it's not a known escape sequence.
Fix parse stuck in loop causing 500 timeout Fixes #3411
2015-07-21 17:12:40 +00:00
},
time.Unix(0, 0)))
Fix regression where measurement names with equals could not be parsed
2015-08-21 17:56:08 +00:00
// measurement, tag and tag value with equals
test(t, `cpu=load,equals\=foo=tag\=value value=1i`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix regression where measurement names with equals could not be parsed
2015-08-21 17:56:08 +00:00
"cpu=load", // Not escaped
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix regression where measurement names with equals could not be parsed
2015-08-21 17:56:08 +00:00
"equals=foo": "tag=value", // Tag and value unescaped
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"value": int64(1),
Fix regression where measurement names with equals could not be parsed
2015-08-21 17:56:08 +00:00
},
time.Unix(0, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
func TestParsePointWithTags(t *testing.T) {
test(t,
"cpu,host=serverA,region=us-east value=1.0 1000000000",
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{"host": "serverA", "region": "us-east"}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{"value": 1.0}, time.Unix(1, 0)))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
Fix the point validation parser to identify and sort tags correctly Fixes #6771.
2016-06-13 14:45:08 +00:00
func TestParsePointWithDuplicateTags(t *testing.T) {
for i, tt := range []struct {
line string
err string
}{
{
line: `cpu,host=serverA,host=serverB value=1i 1000000000`,
err: `unable to parse 'cpu,host=serverA,host=serverB value=1i 1000000000': duplicate tags`,
},
{
line: `cpu,b=2,b=1,c=3 value=1i 1000000000`,
err: `unable to parse 'cpu,b=2,b=1,c=3 value=1i 1000000000': duplicate tags`,
},
{
line: `cpu,b=2,c=3,b=1 value=1i 1000000000`,
err: `unable to parse 'cpu,b=2,c=3,b=1 value=1i 1000000000': duplicate tags`,
},
} {
_, err := models.ParsePointsString(tt.line)
if err == nil || tt.err != err.Error() {
t.Errorf("%d. ParsePoint() expected error '%s'. got '%s'", i, tt.err, err)
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
func TestParsePointWithStringField(t *testing.T) {
test(t, `cpu,host=serverA,region=us-east value=1.0,str="foo",str2="bar" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"value": 1.0,
"str": "foo",
"str2": "bar",
},
time.Unix(1, 0)),
)
test(t, `cpu,host=serverA,region=us-east str="foo \" bar" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint("cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"str": `foo " bar`,
},
time.Unix(1, 0)),
)
}
func TestParsePointWithStringWithSpaces(t *testing.T) {
test(t, `cpu,host=serverA,region=us-east value=1.0,str="foo bar" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"value": 1.0,
"str": "foo bar", // spaces in string value
},
time.Unix(1, 0)),
)
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
Fix parsing string fields with newlines Newlines in a string field would cause the parser to return the line prematurely causing "unbalanced quotes" errors. This makes the line scanning aware of quote fields so that the whole line is returned. Fixes #3545
2015-08-05 20:12:19 +00:00
func TestParsePointWithStringWithNewline(t *testing.T) {
test(t, "cpu,host=serverA,region=us-east value=1.0,str=\"foo\nbar\" 1000000000",
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix parsing string fields with newlines Newlines in a string field would cause the parser to return the line prematurely causing "unbalanced quotes" errors. This makes the line scanning aware of quote fields so that the whole line is returned. Fixes #3545
2015-08-05 20:12:19 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix parsing string fields with newlines Newlines in a string field would cause the parser to return the line prematurely causing "unbalanced quotes" errors. This makes the line scanning aware of quote fields so that the whole line is returned. Fixes #3545
2015-08-05 20:12:19 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix parsing string fields with newlines Newlines in a string field would cause the parser to return the line prematurely causing "unbalanced quotes" errors. This makes the line scanning aware of quote fields so that the whole line is returned. Fixes #3545
2015-08-05 20:12:19 +00:00
"value": 1.0,
"str": "foo\nbar", // newline in string value
},
time.Unix(1, 0)),
)
}
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
func TestParsePointWithStringWithCommas(t *testing.T) {
// escaped comma
test(t, `cpu,host=serverA,region=us-east value=1.0,str="foo\,bar" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
"value": 1.0,
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"str": `foo\,bar`, // commas in string value
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
},
time.Unix(1, 0)),
)
// non-escaped comma
test(t, `cpu,host=serverA,region=us-east value=1.0,str="foo,bar" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
"value": 1.0,
"str": "foo,bar", // commas in string value
},
time.Unix(1, 0)),
Fix string fields w/ trailing slashes A string field w/ a trailing slash before the quote would parse incorrectly because the quote would be seen as escaped. We have to treat \\ as an escape sequence within strings in order to handle this.
2016-12-01 22:24:11 +00:00
)
// string w/ trailing escape chars
test(t, `cpu,host=serverA,region=us-east str="foo\\",str2="bar" 1000000000`,
NewTestPoint(
"cpu",
models.NewTags(map[string]string{
"host": "serverA",
"region": "us-east",
}),
models.Fields{
"str": "foo\\", // trailing escape char
"str2": "bar",
},
time.Unix(1, 0)),
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
)
Fix panic: runtime error: index out of range If the measurement started with a quote, a panic would happen. This is a reegression due to cb7f0b8. This also uncovered that measurement names were being escaped incorrectly. The escape codes for tag and fields also includes `=` and '"` which should not be escaped for measurement names. Fixes #3681
2015-08-18 16:24:03 +00:00
}
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
Fix panic: runtime error: index out of range If the measurement started with a quote, a panic would happen. This is a reegression due to cb7f0b8. This also uncovered that measurement names were being escaped incorrectly. The escape codes for tag and fields also includes `=` and '"` which should not be escaped for measurement names. Fixes #3681
2015-08-18 16:24:03 +00:00
func TestParsePointQuotedMeasurement(t *testing.T) {
// non-escaped comma
test(t, `"cpu",host=serverA,region=us-east value=1.0 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix panic: runtime error: index out of range If the measurement started with a quote, a panic would happen. This is a reegression due to cb7f0b8. This also uncovered that measurement names were being escaped incorrectly. The escape codes for tag and fields also includes `=` and '"` which should not be escaped for measurement names. Fixes #3681
2015-08-18 16:24:03 +00:00
`"cpu"`,
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix panic: runtime error: index out of range If the measurement started with a quote, a panic would happen. This is a reegression due to cb7f0b8. This also uncovered that measurement names were being escaped incorrectly. The escape codes for tag and fields also includes `=` and '"` which should not be escaped for measurement names. Fixes #3681
2015-08-18 16:24:03 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix panic: runtime error: index out of range If the measurement started with a quote, a panic would happen. This is a reegression due to cb7f0b8. This also uncovered that measurement names were being escaped incorrectly. The escape codes for tag and fields also includes `=` and '"` which should not be escaped for measurement names. Fixes #3681
2015-08-18 16:24:03 +00:00
"value": 1.0,
},
time.Unix(1, 0)),
)
Fix parsing commas in string field values Fixes a panic on writes because the field value was not parse correctly. panic: unsupported value type during encode fields: <nil> goroutine 117 [running]: github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 21:44:43 +00:00
}
Ensure tags use the correct escape codes. Similar to measurement names, some characters were being escaped incorrectly.
2015-08-18 16:44:31 +00:00
func TestParsePointQuotedTags(t *testing.T) {
test(t, `cpu,"host"="serverA",region=us-east value=1.0 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Ensure tags use the correct escape codes. Similar to measurement names, some characters were being escaped incorrectly.
2015-08-18 16:44:31 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure tags use the correct escape codes. Similar to measurement names, some characters were being escaped incorrectly.
2015-08-18 16:44:31 +00:00
`"host"`: `"serverA"`,
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure tags use the correct escape codes. Similar to measurement names, some characters were being escaped incorrectly.
2015-08-18 16:44:31 +00:00
"value": 1.0,
},
time.Unix(1, 0)),
)
}
Fix panic: runtime error: slice bounds out of range Fixes #8538
2017-11-09 00:00:25 +00:00
func TestParsePoint_TrailingSlash(t *testing.T) {
_, err := models.ParsePointsString(`a v=1 0\`)
if err == nil {
t.Fatalf("ParsePoints failed: %v", err)
} else if !strings.Contains(err.Error(), "bad timestamp") {
t.Fatalf("ParsePoints unexpected error: %v", err)
}
}
Fix write fails for multiple points when tag starts with quote Fixes #3928
2015-09-01 16:56:31 +00:00
func TestParsePointsUnbalancedQuotedTags(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsString("baz,mytag=\"a x=1 1441103862125\nbaz,mytag=a z=1 1441103862126")
Fix write fails for multiple points when tag starts with quote Fixes #3928
2015-09-01 16:56:31 +00:00
if err != nil {
t.Fatalf("ParsePoints failed: %v", err)
}
if exp := 2; len(pts) != exp {
t.Fatalf("ParsePoints count mismatch. got %v, exp %v", len(pts), exp)
}
// Expected " in the tag value
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
exp := models.MustNewPoint("baz", models.NewTags(map[string]string{"mytag": `"a`}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{"x": float64(1)}, time.Unix(0, 1441103862125))
Fix write fails for multiple points when tag starts with quote Fixes #3928
2015-09-01 16:56:31 +00:00
if pts[0].String() != exp.String() {
t.Errorf("Point mismatch:\ngot: %v\nexp: %v", pts[0].String(), exp.String())
}
// Expected two points to ensure we did not overscan the line
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
exp = models.MustNewPoint("baz", models.NewTags(map[string]string{"mytag": `a`}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{"z": float64(1)}, time.Unix(0, 1441103862126))
Fix write fails for multiple points when tag starts with quote Fixes #3928
2015-09-01 16:56:31 +00:00
if pts[1].String() != exp.String() {
t.Errorf("Point mismatch:\ngot: %v\nexp: %v", pts[1].String(), exp.String())
}
}
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
func TestParsePointEscapedStringsAndCommas(t *testing.T) {
// non-escaped comma and quotes
test(t, `cpu,host=serverA,region=us-east value="{Hello\"{,}\" World}" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"value": `{Hello"{,}" World}`,
},
time.Unix(1, 0)),
)
// escaped comma and quotes
test(t, `cpu,host=serverA,region=us-east value="{Hello\"{\,}\" World}" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix string field value escaping Commas and quotes could get escaped and parsed incorrectly if they were both present in a string value. Fixes #3013
2015-06-22 18:22:03 +00:00
"value": `{Hello"{\,}" World}`,
},
time.Unix(1, 0)),
)
}
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
func TestParsePointWithStringWithEquals(t *testing.T) {
Return error when parsing fields with no name Fixes #3061
2015-06-26 18:02:46 +00:00
test(t, `cpu,host=serverA,region=us-east str="foo=bar",value=1.0 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": 1.0,
"str": "foo=bar", // spaces in string value
},
time.Unix(1, 0)),
)
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
func TestParsePointWithStringWithBackslash(t *testing.T) {
test(t, `cpu value="test\\\"" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"value": `test\"`,
},
time.Unix(1, 0)),
)
test(t, `cpu value="test\\" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"value": `test\`,
},
time.Unix(1, 0)),
)
test(t, `cpu value="test\\\"" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"value": `test\"`,
},
time.Unix(1, 0)),
)
test(t, `cpu value="test\"" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix inserting string values with backslashes Fixes #3682
2015-08-19 15:27:17 +00:00
"value": `test"`,
},
time.Unix(1, 0)),
)
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
func TestParsePointWithBoolField(t *testing.T) {
Fix regression parsing boolean True/False values
2015-07-13 17:27:24 +00:00
test(t, `cpu,host=serverA,region=us-east true=true,t=t,T=T,TRUE=TRUE,True=True,false=false,f=f,F=F,FALSE=FALSE,False=False 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Be more explicit about boolean value parsing Supported boolean values are now t, T, true, TRUE, f, F, false, and FALSE. This is what the strconv.ParseBool function supports with the exception of 1 and 0. 1 and 0 would be parsed as ints in the line protocol. Previously, any non-true value would be parsed as false. e.g. value=blah would parse to false. This will now return an error as parsing time.
2015-06-11 20:53:53 +00:00
"t": true,
"T": true,
"true": true,
Fix regression parsing boolean True/False values
2015-07-13 17:27:24 +00:00
"True": true,
Be more explicit about boolean value parsing Supported boolean values are now t, T, true, TRUE, f, F, false, and FALSE. This is what the strconv.ParseBool function supports with the exception of 1 and 0. 1 and 0 would be parsed as ints in the line protocol. Previously, any non-true value would be parsed as false. e.g. value=blah would parse to false. This will now return an error as parsing time.
2015-06-11 20:53:53 +00:00
"TRUE": true,
"f": false,
"F": false,
"false": false,
Fix regression parsing boolean True/False values
2015-07-13 17:27:24 +00:00
"False": false,
Be more explicit about boolean value parsing Supported boolean values are now t, T, true, TRUE, f, F, false, and FALSE. This is what the strconv.ParseBool function supports with the exception of 1 and 0. 1 and 0 would be parsed as ints in the line protocol. Previously, any non-true value would be parsed as false. e.g. value=blah would parse to false. This will now return an error as parsing time.
2015-06-11 20:53:53 +00:00
"FALSE": false,
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
},
time.Unix(1, 0)),
)
}
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
func TestParsePointUnicodeString(t *testing.T) {
test(t, `cpu,host=serverA,region=us-east value="wè" 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"host": "serverA",
"region": "us-east",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
"value": "wè",
},
time.Unix(1, 0)),
)
}
Handle negative timestamps
2015-10-19 17:40:18 +00:00
func TestParsePointNegativeTimestamp(t *testing.T) {
test(t, `cpu value=1 -1`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Handle negative timestamps
2015-10-19 17:40:18 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
Handle negative timestamps
2015-10-19 17:40:18 +00:00
models.Fields{
"value": 1.0,
},
time.Unix(0, -1)),
)
}
func TestParsePointMaxTimestamp(t *testing.T) {
Modify the max nanosecond time to be one nanosecond less The highest time represented by a nanosecond needs to be used for an exclusive range, so the maximum time needs to be one less than the possible maximum number of nanoseconds representable by an int64 so that we don't lose a point at that one time. Previously worked in the open source version because the timestamp used for finding a shard would be truncated by the retention policy so the lookup time didn't run into this edge case because it didn't rest on the truncation boundary. Since that point didn't really belong in that shard group and was placed there by mistake, it's best to fix this bug since the timestamp used to create the shard group should be capable of retrieving it.
2016-06-16 17:15:41 +00:00
test(t, fmt.Sprintf(`cpu value=1 %d`, models.MaxNanoTime),
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Handle negative timestamps
2015-10-19 17:40:18 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
Handle negative timestamps
2015-10-19 17:40:18 +00:00
models.Fields{
"value": 1.0,
},
Reject timestamps too far in future
2016-05-19 15:37:15 +00:00
time.Unix(0, models.MaxNanoTime)),
Handle negative timestamps
2015-10-19 17:40:18 +00:00
)
}
func TestParsePointMinTimestamp(t *testing.T) {
Support negative timestamps in the query engine Negative timestamps are now supported. We also now refuse two nanoseconds that are at the edge of the minimum time window. One of the nanoseconds we do not accept is because we need MinInt64 to be used for some internal comparisons in the TSM engine and it was causing an underflow when we subtracted one from the minimum time. The second is so we can have one minimum time that signifies the default minimum that nobody can write to (so we can implicitly rewrite the timestamp on aggregate queries) but still use the explicit timestamp if it is given to us by the user. We aren't able to tell the difference between if the user provided it or if it was implicit without those values being different. If the default minimum time is used with an aggregate query, we rewrite the time to be the epoch for backwards compatibility since we believe that's more important than supporting that extra nanosecond.
2016-08-25 17:52:39 +00:00
test(t, `cpu value=1 -9223372036854775806`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Handle negative timestamps
2015-10-19 17:40:18 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
Handle negative timestamps
2015-10-19 17:40:18 +00:00
models.Fields{
"value": 1.0,
},
Reject timestamps too far in future
2016-05-19 15:37:15 +00:00
time.Unix(0, models.MinNanoTime)),
Handle negative timestamps
2015-10-19 17:40:18 +00:00
)
}
func TestParsePointInvalidTimestamp(t *testing.T) {
Ensure points with trailing whitespace are accepted
2016-05-19 15:58:18 +00:00
examples := []string{
"cpu value=1 9223372036854775808",
"cpu value=1 -92233720368547758078",
"cpu value=1 -",
"cpu value=1 -/",
"cpu value=1 -1?",
"cpu value=1 1-",
"cpu value=1 9223372036854775807 12",
Handle negative timestamps
2015-10-19 17:40:18 +00:00
}
Ensure points with trailing whitespace are accepted
2016-05-19 15:58:18 +00:00
for i, example := range examples {
_, err := models.ParsePointsString(example)
if err == nil {
t.Fatalf("[Example %d] ParsePoints failed: %v", i, err)
}
Add test for 1- timestamp
2015-10-19 20:06:13 +00:00
}
Handle negative timestamps
2015-10-19 17:40:18 +00:00
}
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
func TestNewPointFloatWithoutDecimal(t *testing.T) {
test(t, `cpu value=1 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
"value": 1.0,
},
time.Unix(1, 0)),
)
}
Fix parsing negative floats Fixes #2919
2015-06-12 06:06:54 +00:00
func TestNewPointNegativeFloat(t *testing.T) {
test(t, `cpu value=-0.64 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix parsing negative floats Fixes #2919
2015-06-12 06:06:54 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix parsing negative floats Fixes #2919
2015-06-12 06:06:54 +00:00
"value": -0.64,
},
time.Unix(1, 0)),
)
}
func TestNewPointFloatNoDecimal(t *testing.T) {
test(t, `cpu value=1. 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix parsing negative floats Fixes #2919
2015-06-12 06:06:54 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix parsing negative floats Fixes #2919
2015-06-12 06:06:54 +00:00
"value": 1.0,
},
time.Unix(1, 0)),
)
}
Fix large integers getting converted to floats during remote writes Fixes #2960 Integers were were written back to line protocol using strconv.FormatFloat incorrectly. Large integers are written in scientific notation which causes their type to change to a float when parsed back.
2015-06-12 17:08:11 +00:00
func TestNewPointFloatScientific(t *testing.T) {
test(t, `cpu value=6.632243e+06 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix large integers getting converted to floats during remote writes Fixes #2960 Integers were were written back to line protocol using strconv.FormatFloat incorrectly. Large integers are written in scientific notation which causes their type to change to a float when parsed back.
2015-06-12 17:08:11 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fix large integers getting converted to floats during remote writes Fixes #2960 Integers were were written back to line protocol using strconv.FormatFloat incorrectly. Large integers are written in scientific notation which causes their type to change to a float when parsed back.
2015-06-12 17:08:11 +00:00
"value": float64(6632243),
},
time.Unix(1, 0)),
)
}
func TestNewPointLargeInteger(t *testing.T) {
integers in line protocol now have to end with `i`
2015-07-31 01:51:18 +00:00
test(t, `cpu value=6632243i 1000000000`,
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
NewTestPoint(
Fix large integers getting converted to floats during remote writes Fixes #2960 Integers were were written back to line protocol using strconv.FormatFloat incorrectly. Large integers are written in scientific notation which causes their type to change to a float when parsed back.
2015-06-12 17:08:11 +00:00
"cpu",
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
models.NewTags(map[string]string{}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{
Fixes #5380 and #5381
2016-01-17 19:31:01 +00:00
"value": int64(6632243), // if incorrectly encoded as a float, it would show up as 6.632243e+06
Fix large integers getting converted to floats during remote writes Fixes #2960 Integers were were written back to line protocol using strconv.FormatFloat incorrectly. Large integers are written in scientific notation which causes their type to change to a float when parsed back.
2015-06-12 17:08:11 +00:00
},
time.Unix(1, 0)),
)
}
Return parse error for NaN float values Not currently supported.
2015-10-26 22:05:27 +00:00
func TestParsePointNaN(t *testing.T) {
_, err := models.ParsePointsString("cpu value=NaN 1000000000")
if err == nil {
t.Fatalf("ParsePoints expected error, got nil")
}
Parse NaN as float Fixes #3230
2015-07-06 16:18:07 +00:00
Return parse error for NaN float values Not currently supported.
2015-10-26 22:05:27 +00:00
_, err = models.ParsePointsString("cpu value=nAn 1000000000")
if err == nil {
t.Fatalf("ParsePoints expected error, got nil")
}
Fix parsing NaN values without timestamps Fixes #3539 partially. NaN cannot be queried though and needs to be handled by the query engine differently.
2015-08-21 18:12:48 +00:00
Return parse error for NaN float values Not currently supported.
2015-10-26 22:05:27 +00:00
_, err = models.ParsePointsString("cpu value=NaN")
if err == nil {
t.Fatalf("ParsePoints expected error, got nil")
}
Fix panic parsing measurement with large number of tags Defaults to handling measurements with up to 100 tags and will now grow the slice if there are more instead of panicing. Fixes #3511
2015-08-05 18:24:42 +00:00
}
func TestNewPointLargeNumberOfTags(t *testing.T) {
tags := ""
for i := 0; i < 255; i++ {
tags += fmt.Sprintf(",tag%d=value%d", i, i)
}
Parse NaN as float Fixes #3230
2015-07-06 16:18:07 +00:00
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pt, err := models.ParsePointsString(fmt.Sprintf("cpu%s value=1", tags))
Fix panic parsing measurement with large number of tags Defaults to handling measurements with up to 100 tags and will now grow the slice if there are more instead of panicing. Fixes #3511
2015-08-05 18:24:42 +00:00
if err != nil {
t.Fatalf("ParsePoints() with max tags failed: %v", err)
}
if len(pt[0].Tags()) != 255 {
Add extra test coverage; fix test typo
2015-11-18 23:18:28 +00:00
t.Fatalf("expected %d tags, got %d", 255, len(pt[0].Tags()))
Fix panic parsing measurement with large number of tags Defaults to handling measurements with up to 100 tags and will now grow the slice if there are more instead of panicing. Fixes #3511
2015-08-05 18:24:42 +00:00
}
Parse NaN as float Fixes #3230
2015-07-06 16:18:07 +00:00
}
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
func TestParsePointIntsFloats(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePoints([]byte(`cpu,host=serverA,region=us-east int=10i,float=11.0,float2=12.1 1000000000`))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
if err != nil {
t.Fatalf(`ParsePoints() failed. got %s`, err)
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
if exp := 1; len(pts) != exp {
t.Errorf("ParsePoint() len mismatch: got %v, exp %v", len(pts), exp)
}
pt := pts[0]
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := pt.Fields()
if err != nil {
t.Fatal(err)
}
if _, ok := fields["int"].(int64); !ok {
t.Errorf("ParsePoint() int field mismatch: got %T, exp %T", fields["int"], int64(10))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
if _, ok := fields["float"].(float64); !ok {
t.Errorf("ParsePoint() float field mismatch: got %T, exp %T", fields["float64"], float64(11.0))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
if _, ok := fields["float2"].(float64); !ok {
t.Errorf("ParsePoint() float field mismatch: got %T, exp %T", fields["float64"], float64(12.1))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
func TestParsePointKeyUnsorted(t *testing.T) {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePoints([]byte("cpu,last=1,first=2 value=1i"))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
if err != nil {
t.Fatalf(`ParsePoints() failed. got %s`, err)
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
if exp := 1; len(pts) != exp {
t.Errorf("ParsePoint() len mismatch: got %v, exp %v", len(pts), exp)
}
pt := pts[0]
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
if exp := "cpu,first=2,last=1"; string(pt.Key()) != exp {
Add extra test coverage; fix test typo
2015-11-18 23:18:28 +00:00
t.Errorf("ParsePoint key not sorted. got %v, exp %v", string(pt.Key()), exp)
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
}
}
func TestParsePointToString(t *testing.T) {
support all number types when decoding a point
2015-08-25 13:49:49 +00:00
line := `cpu,host=serverA,region=us-east bool=false,float=11,float2=12.123,int=10i,str="string val" 1000000000`
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePoints([]byte(line))
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
if err != nil {
t.Fatalf(`ParsePoints() failed. got %s`, err)
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
if exp := 1; len(pts) != exp {
t.Errorf("ParsePoint() len mismatch: got %v, exp %v", len(pts), exp)
}
pt := pts[0]
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
got := pt.String()
if line != got {
t.Errorf("ParsePoint() to string mismatch:\n got %v\n exp %v", got, line)
}
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
pt = models.MustNewPoint("cpu", models.NewTags(map[string]string{"host": "serverA", "region": "us-east"}),
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
models.Fields{"int": 10, "float": float64(11.0), "float2": float64(12.123), "bool": false, "str": "string val"},
Add text protocol parsing and serialzation for points This changes the implementation of point to minimize the extra processing needed to parse and marshal point data though the system.
2015-05-28 21:47:52 +00:00
time.Unix(1, 0))
got = pt.String()
if line != got {
t.Errorf("NewPoint() to string mismatch:\n got %v\n exp %v", got, line)
Move data.Point to tsdb.Point
2015-05-22 21:00:51 +00:00
}
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
func TestParsePointsWithPrecision(t *testing.T) {
make line protocol round or parse precision as expected
2015-06-04 17:22:53 +00:00
tests := []struct {
name string
line string
precision string
exp string
}{
{
name: "nanosecond by default",
line: `cpu,host=serverA,region=us-east value=1.0 946730096789012345`,
precision: "",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
},
{
name: "nanosecond",
line: `cpu,host=serverA,region=us-east value=1.0 946730096789012345`,
precision: "n",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
},
{
name: "microsecond",
line: `cpu,host=serverA,region=us-east value=1.0 946730096789012`,
precision: "u",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012000",
},
{
name: "millisecond",
line: `cpu,host=serverA,region=us-east value=1.0 946730096789`,
precision: "ms",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789000000",
},
{
name: "second",
line: `cpu,host=serverA,region=us-east value=1.0 946730096`,
precision: "s",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096000000000",
},
{
name: "minute",
line: `cpu,host=serverA,region=us-east value=1.0 15778834`,
precision: "m",
exp: "cpu,host=serverA,region=us-east value=1.0 946730040000000000",
},
{
name: "hour",
line: `cpu,host=serverA,region=us-east value=1.0 262980`,
precision: "h",
exp: "cpu,host=serverA,region=us-east value=1.0 946728000000000000",
},
}
for _, test := range tests {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsWithPrecision([]byte(test.line), time.Now().UTC(), test.precision)
make line protocol round or parse precision as expected
2015-06-04 17:22:53 +00:00
if err != nil {
t.Fatalf(`%s: ParsePoints() failed. got %s`, test.name, err)
}
if exp := 1; len(pts) != exp {
t.Errorf("%s: ParsePoint() len mismatch: got %v, exp %v", test.name, len(pts), exp)
}
pt := pts[0]
got := pt.String()
if got != test.exp {
t.Errorf("%s: ParsePoint() to string mismatch:\n got %v\n exp %v", test.name, got, test.exp)
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
}
make line protocol round or parse precision as expected
2015-06-04 17:22:53 +00:00
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
make line protocol round or parse precision as expected
2015-06-04 17:22:53 +00:00
func TestParsePointsWithPrecisionNoTime(t *testing.T) {
line := `cpu,host=serverA,region=us-east value=1.0`
tm, _ := time.Parse(time.RFC3339Nano, "2000-01-01T12:34:56.789012345Z")
tests := []struct {
name string
precision string
exp string
}{
{
name: "no precision",
precision: "",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
},
{
name: "nanosecond precision",
precision: "n",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
},
{
name: "microsecond precision",
precision: "u",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012000",
},
{
name: "millisecond precision",
precision: "ms",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789000000",
},
{
name: "second precision",
precision: "s",
exp: "cpu,host=serverA,region=us-east value=1.0 946730096000000000",
},
{
name: "minute precision",
precision: "m",
exp: "cpu,host=serverA,region=us-east value=1.0 946730040000000000",
},
{
name: "hour precision",
precision: "h",
exp: "cpu,host=serverA,region=us-east value=1.0 946728000000000000",
},
}
for _, test := range tests {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsWithPrecision([]byte(line), tm, test.precision)
make line protocol round or parse precision as expected
2015-06-04 17:22:53 +00:00
if err != nil {
t.Fatalf(`%s: ParsePoints() failed. got %s`, test.name, err)
}
if exp := 1; len(pts) != exp {
t.Errorf("%s: ParsePoint() len mismatch: got %v, exp %v", test.name, len(pts), exp)
}
pt := pts[0]
got := pt.String()
if got != test.exp {
t.Errorf("%s: ParsePoint() to string mismatch:\n got %v\n exp %v", test.name, got, test.exp)
}
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 17:16:59 +00:00
}
}
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
allow comments in line protocol
2015-06-18 18:59:20 +00:00
func TestParsePointsWithPrecisionComments(t *testing.T) {
tests := []struct {
name string
batch string
exp string
lenPoints int
}{
{
name: "comment only",
batch: `# comment only`,
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
lenPoints: 0,
},
{
name: "point with comment above",
batch: `# a point is below
cpu,host=serverA,region=us-east value=1.0 946730096789012345`,
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
lenPoints: 1,
},
{
name: "point with comment below",
batch: `cpu,host=serverA,region=us-east value=1.0 946730096789012345
# end of points`,
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
lenPoints: 1,
},
{
name: "indented comment",
batch: ` # a point is below
cpu,host=serverA,region=us-east value=1.0 946730096789012345`,
exp: "cpu,host=serverA,region=us-east value=1.0 946730096789012345",
lenPoints: 1,
},
}
for _, test := range tests {
refactor Points and Rows to dedicated packages
2015-09-16 20:33:08 +00:00
pts, err := models.ParsePointsWithPrecision([]byte(test.batch), time.Now().UTC(), "")
allow comments in line protocol
2015-06-18 18:59:20 +00:00
if err != nil {
t.Fatalf(`%s: ParsePoints() failed. got %s`, test.name, err)
}
pointsLength := len(pts)
if exp := test.lenPoints; pointsLength != exp {
t.Errorf("%s: ParsePoint() len mismatch: got %v, exp %v", test.name, pointsLength, exp)
}
if pointsLength > 0 {
pt := pts[0]
got := pt.String()
if got != test.exp {
t.Errorf("%s: ParsePoint() to string mismatch:\n got %v\n exp %v", test.name, got, test.exp)
}
}
}
}
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
func TestNewPointEscaped(t *testing.T) {
// commas
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
pt := models.MustNewPoint("cpu,main", models.NewTags(map[string]string{"tag,bar": "value"}), models.Fields{"name,bar": 1.0}, time.Unix(0, 0))
support all number types when decoding a point
2015-08-25 13:49:49 +00:00
if exp := `cpu\,main,tag\,bar=value name\,bar=1 0`; pt.String() != exp {
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
t.Errorf("NewPoint().String() mismatch.\ngot %v\nexp %v", pt.String(), exp)
}
// spaces
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
pt = models.MustNewPoint("cpu main", models.NewTags(map[string]string{"tag bar": "value"}), models.Fields{"name bar": 1.0}, time.Unix(0, 0))
support all number types when decoding a point
2015-08-25 13:49:49 +00:00
if exp := `cpu\ main,tag\ bar=value name\ bar=1 0`; pt.String() != exp {
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
t.Errorf("NewPoint().String() mismatch.\ngot %v\nexp %v", pt.String(), exp)
}
// equals
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
pt = models.MustNewPoint("cpu=main", models.NewTags(map[string]string{"tag=bar": "value=foo"}), models.Fields{"name=bar": 1.0}, time.Unix(0, 0))
support all number types when decoding a point
2015-08-25 13:49:49 +00:00
if exp := `cpu=main,tag\=bar=value\=foo name\=bar=1 0`; pt.String() != exp {
Ensure comma,space and equals are escaped
2015-05-29 20:34:06 +00:00
t.Errorf("NewPoint().String() mismatch.\ngot %v\nexp %v", pt.String(), exp)
}
}
Serialized unknown types as strings When creating a point manually, the field values are interface{} which allows unsupported types to be passed in. Previously, the code would panic. It will now default to string representation of the value if it's not a known type.
2015-07-06 22:03:03 +00:00
check point without fields when NewPoint
2015-11-17 05:21:52 +00:00
func TestNewPointWithoutField(t *testing.T) {
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
_, err := models.NewPoint("cpu", models.NewTags(map[string]string{"tag": "bar"}), models.Fields{}, time.Unix(0, 0))
check point without fields when NewPoint
2015-11-17 05:21:52 +00:00
if err == nil {
t.Fatalf(`NewPoint() expected error. got nil`)
}
}
Serialized unknown types as strings When creating a point manually, the field values are interface{} which allows unsupported types to be passed in. Previously, the code would panic. It will now default to string representation of the value if it's not a known type.
2015-07-06 22:03:03 +00:00
func TestNewPointUnhandledType(t *testing.T) {
// nil value
Prevent creating points with NaN float values Float values are not supported in the existing engine and the tsm1 engines. This changes NewPoint to return an error if a field value contains a NaN field. It also allows us to validate fields to prevent other unsupported types from sneaking in through other input plugins.
2015-10-27 16:21:54 +00:00
pt := models.MustNewPoint("cpu", nil, models.Fields{"value": nil}, time.Unix(0, 0))
Serialized unknown types as strings When creating a point manually, the field values are interface{} which allows unsupported types to be passed in. Previously, the code would panic. It will now default to string representation of the value if it's not a known type.
2015-07-06 22:03:03 +00:00
if exp := `cpu value= 0`; pt.String() != exp {
t.Errorf("NewPoint().String() mismatch.\ngot %v\nexp %v", pt.String(), exp)
}
// unsupported type gets stored as string
Use UTC for test assertion data
2015-07-07 03:11:09 +00:00
now := time.Unix(0, 0).UTC()
Prevent creating points with NaN float values Float values are not supported in the existing engine and the tsm1 engines. This changes NewPoint to return an error if a field value contains a NaN field. It also allows us to validate fields to prevent other unsupported types from sneaking in through other input plugins.
2015-10-27 16:21:54 +00:00
pt = models.MustNewPoint("cpu", nil, models.Fields{"value": now}, time.Unix(0, 0))
Use UTC for test assertion data
2015-07-07 03:11:09 +00:00
if exp := `cpu value="1970-01-01 00:00:00 +0000 UTC" 0`; pt.String() != exp {
Serialized unknown types as strings When creating a point manually, the field values are interface{} which allows unsupported types to be passed in. Previously, the code would panic. It will now default to string representation of the value if it's not a known type.
2015-07-06 22:03:03 +00:00
t.Errorf("NewPoint().String() mismatch.\ngot %v\nexp %v", pt.String(), exp)
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := pt.Fields()
if err != nil {
t.Fatal(err)
}
if exp := "1970-01-01 00:00:00 +0000 UTC"; fields["value"] != exp {
Serialized unknown types as strings When creating a point manually, the field values are interface{} which allows unsupported types to be passed in. Previously, the code would panic. It will now default to string representation of the value if it's not a known type.
2015-07-06 22:03:03 +00:00
t.Errorf("NewPoint().String() mismatch.\ngot %v\nexp %v", pt.String(), exp)
}
}
Fix measurement name being double-escaped during replication Fixes #3708 #3704
2015-08-18 16:02:48 +00:00
func TestMakeKeyEscaped(t *testing.T) {
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
if exp, got := `cpu\ load`, models.MakeKey([]byte(`cpu\ load`), models.NewTags(map[string]string{})); string(got) != exp {
Fix measurement name being double-escaped during replication Fixes #3708 #3704
2015-08-18 16:02:48 +00:00
t.Errorf("MakeKey() mismatch.\ngot %v\nexp %v", got, exp)
}
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
if exp, got := `cpu\ load`, models.MakeKey([]byte(`cpu load`), models.NewTags(map[string]string{})); string(got) != exp {
Fix measurement name being double-escaped during replication Fixes #3708 #3704
2015-08-18 16:02:48 +00:00
t.Errorf("MakeKey() mismatch.\ngot %v\nexp %v", got, exp)
}
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
if exp, got := `cpu\,load`, models.MakeKey([]byte(`cpu\,load`), models.NewTags(map[string]string{})); string(got) != exp {
Fix measurement name being double-escaped during replication Fixes #3708 #3704
2015-08-18 16:02:48 +00:00
t.Errorf("MakeKey() mismatch.\ngot %v\nexp %v", got, exp)
}
reduce memory allocations in index This commit changes the index to point to index data in the shards instead of keeping it in-memory on the heap.
2016-06-30 16:49:53 +00:00
if exp, got := `cpu\,load`, models.MakeKey([]byte(`cpu,load`), models.NewTags(map[string]string{})); string(got) != exp {
Fix measurement name being double-escaped during replication Fixes #3708 #3704
2015-08-18 16:02:48 +00:00
t.Errorf("MakeKey() mismatch.\ngot %v\nexp %v", got, exp)
}
}
Add function to tsdb.point to get line-protocol string in the correct units
2015-09-15 17:59:52 +00:00
func TestPrecisionString(t *testing.T) {
tags := map[string]interface{}{"value": float64(1)}
tm, _ := time.Parse(time.RFC3339Nano, "2000-01-01T12:34:56.789012345Z")
tests := []struct {
name string
precision string
exp string
}{
{
name: "no precision",
precision: "",
exp: "cpu value=1 946730096789012345",
},
{
name: "nanosecond precision",
precision: "ns",
exp: "cpu value=1 946730096789012345",
},
{
name: "microsecond precision",
precision: "u",
exp: "cpu value=1 946730096789012",
},
{
name: "millisecond precision",
precision: "ms",
exp: "cpu value=1 946730096789",
},
{
name: "second precision",
precision: "s",
exp: "cpu value=1 946730096",
},
{
name: "minute precision",
precision: "m",
exp: "cpu value=1 15778834",
},
{
name: "hour precision",
precision: "h",
exp: "cpu value=1 262980",
},
}
for _, test := range tests {
Prevent creating points with NaN float values Float values are not supported in the existing engine and the tsm1 engines. This changes NewPoint to return an error if a field value contains a NaN field. It also allows us to validate fields to prevent other unsupported types from sneaking in through other input plugins.
2015-10-27 16:21:54 +00:00
pt := models.MustNewPoint("cpu", nil, tags, tm)
Add function to tsdb.point to get line-protocol string in the correct units
2015-09-15 17:59:52 +00:00
act := pt.PrecisionString(test.precision)
if act != test.exp {
t.Errorf("%s: PrecisionString() mismatch:\n actual: %v\n exp: %v",
test.name, act, test.exp)
}
}
}
skip empty string for start position
2015-10-28 14:12:53 +00:00
Implement a UDP client closes #4647
2015-11-03 20:20:52 +00:00
func TestRoundedString(t *testing.T) {
tags := map[string]interface{}{"value": float64(1)}
tm, _ := time.Parse(time.RFC3339Nano, "2000-01-01T12:34:56.789012345Z")
tests := []struct {
name string
precision time.Duration
exp string
}{
{
name: "no precision",
precision: time.Duration(0),
exp: "cpu value=1 946730096789012345",
},
{
name: "nanosecond precision",
precision: time.Nanosecond,
exp: "cpu value=1 946730096789012345",
},
{
name: "microsecond precision",
precision: time.Microsecond,
exp: "cpu value=1 946730096789012000",
},
{
name: "millisecond precision",
precision: time.Millisecond,
exp: "cpu value=1 946730096789000000",
},
{
name: "second precision",
precision: time.Second,
exp: "cpu value=1 946730097000000000",
},
{
name: "minute precision",
precision: time.Minute,
exp: "cpu value=1 946730100000000000",
},
{
name: "hour precision",
precision: time.Hour,
exp: "cpu value=1 946731600000000000",
},
}
for _, test := range tests {
pt := models.MustNewPoint("cpu", nil, tags, tm)
act := pt.RoundedString(test.precision)
if act != test.exp {
t.Errorf("%s: RoundedString() mismatch:\n actual: %v\n exp: %v",
test.name, act, test.exp)
}
}
}
skip empty string for start position
2015-10-28 14:12:53 +00:00
func TestParsePointsStringWithExtraBuffer(t *testing.T) {
b := make([]byte, 70*5000)
buf := bytes.NewBuffer(b)
key := "cpu,host=A,region=uswest"
buf.WriteString(fmt.Sprintf("%s value=%.3f 1\n", key, rand.Float64()))
points, err := models.ParsePointsString(buf.String())
if err != nil {
t.Fatalf("failed to write points: %s", err.Error())
}
pointKey := string(points[0].Key())
if len(key) != len(pointKey) {
t.Fatalf("expected length of both keys are same but got %d and %d", len(key), len(pointKey))
}
if key != pointKey {
t.Fatalf("expected both keys are same but got %s and %s", key, pointKey)
}
}
Fix the line protocol scanner to read field keys with quotes correctly Quotes are not supposed to be significant in field keys, but are significant in field values. The code as it currently was would consider quotes in a key to be significant, but the later parser that would unmarshal the fields from the byte string did not consider those quotes to be significant. This meant that the following string: "a=1 The line protocol parser would see a mismatched quote instead of a valid input to the line protocol. But more nefariously, the following string: "a=1"=2 The line protocol parser would ignore the first equals since it is located in the quotation marks and think this was a valid input. It would then pass it on to the field parser who would panic and die when it tried to parse `1"=2` as a number. Fixes #4076.
2015-12-17 20:38:27 +00:00
func TestParsePointsQuotesInFieldKey(t *testing.T) {
Fix scanLine to handle quotes properly Quotes are handled differently in the line protocol depending on when they are encountered. Quotes in field values matter, quotes anywhere else don't. `scanLine()` didn't understand this difference and treated all quotes the same as ones for tag values. This resulted in `scanLine()` reading the wrong amount of data sometimes when quotes were involved. This fixes #5204.
2015-12-29 19:35:00 +00:00
buf := `cpu "a=1
cpu value=2 1`
Fix the line protocol scanner to read field keys with quotes correctly Quotes are not supposed to be significant in field keys, but are significant in field values. The code as it currently was would consider quotes in a key to be significant, but the later parser that would unmarshal the fields from the byte string did not consider those quotes to be significant. This meant that the following string: "a=1 The line protocol parser would see a mismatched quote instead of a valid input to the line protocol. But more nefariously, the following string: "a=1"=2 The line protocol parser would ignore the first equals since it is located in the quotation marks and think this was a valid input. It would then pass it on to the field parser who would panic and die when it tried to parse `1"=2` as a number. Fixes #4076.
2015-12-17 20:38:27 +00:00
points, err := models.ParsePointsString(buf)
if err != nil {
t.Fatalf("failed to write points: %s", err.Error())
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fields, err := points[0].Fields()
if err != nil {
t.Fatal(err)
}
value, ok := fields["\"a"]
Fix the line protocol scanner to read field keys with quotes correctly Quotes are not supposed to be significant in field keys, but are significant in field values. The code as it currently was would consider quotes in a key to be significant, but the later parser that would unmarshal the fields from the byte string did not consider those quotes to be significant. This meant that the following string: "a=1 The line protocol parser would see a mismatched quote instead of a valid input to the line protocol. But more nefariously, the following string: "a=1"=2 The line protocol parser would ignore the first equals since it is located in the quotation marks and think this was a valid input. It would then pass it on to the field parser who would panic and die when it tried to parse `1"=2` as a number. Fixes #4076.
2015-12-17 20:38:27 +00:00
if !ok {
t.Fatalf("expected to parse field '\"a'")
}
if value != float64(1) {
t.Fatalf("expected field value to be 1, got %v", value)
}
// The following input should not parse
buf = `cpu "\, '= "\ v=1.0`
_, err = models.ParsePointsString(buf)
if err == nil {
t.Fatalf("expected parsing failure but got no error")
}
}
Fix scanLine to handle quotes properly Quotes are handled differently in the line protocol depending on when they are encountered. Quotes in field values matter, quotes anywhere else don't. `scanLine()` didn't understand this difference and treated all quotes the same as ones for tag values. This resulted in `scanLine()` reading the wrong amount of data sometimes when quotes were involved. This fixes #5204.
2015-12-29 19:35:00 +00:00
func TestParsePointsQuotesInTags(t *testing.T) {
buf := `t159,label=hey\ "ya a=1i,value=0i
t159,label=another a=2i,value=1i 1`
points, err := models.ParsePointsString(buf)
if err != nil {
t.Fatalf("failed to write points: %s", err.Error())
}
if len(points) != 2 {
t.Fatalf("expected 2 points, got %d", len(points))
}
}
models: add tests for point construction These tests check that NewPoint and NewPointFromBytes return an error if: * arguments specify a point with no fields * arguments specify a point with a field that has an empty name These tests also check that Point.Fields() always returns, even in the presence of corrupt binary data read with NewPointFromBytes. These tests fail at this commit and are fixed by a subsequent commit. Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-02-17 15:47:29 +00:00
Allow blank lines in the line protocol input
2016-08-30 14:25:53 +00:00
func TestParsePointsBlankLine(t *testing.T) {
buf := `cpu value=1i 1000000000
cpu value=2i 2000000000`
points, err := models.ParsePointsString(buf)
if err != nil {
t.Fatalf("failed to write points: %s", err.Error())
}
if len(points) != 2 {
t.Fatalf("expected 2 points, got %d", len(points))
}
}
models: add tests for point construction These tests check that NewPoint and NewPointFromBytes return an error if: * arguments specify a point with no fields * arguments specify a point with a field that has an empty name These tests also check that Point.Fields() always returns, even in the presence of corrupt binary data read with NewPointFromBytes. These tests fail at this commit and are fixed by a subsequent commit. Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-02-17 15:47:29 +00:00
func TestNewPointsWithBytesWithCorruptData(t *testing.T) {
models: improve handling of points with empty field names or with no fields Influx does not support fields with empty names or points with no fields. NewPoint is changed to validate that all field names are non-empty. AddField is removed because we now require that all fields are specified on construction. NewPointFromByte is changed to return an error if a unmarshaled binary point does not have any fields. newFieldsFromBinary is changed to prevent an infinite loop that can arise while attempting to parse corrupt binary point data. TestNewPointsWithBytesWithCorruptData is changed to reflect the change in the behaviour of NewPointFromByte. Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-02-20 11:22:26 +00:00
corrupted := []byte{0, 0, 0, 3, 102, 111, 111, 0, 0, 0, 4, 61, 34, 65, 34, 1, 0, 0, 0, 14, 206, 86, 119, 24, 32, 72, 233, 168, 2, 148}
p, err := models.NewPointFromBytes(corrupted)
if p != nil || err == nil {
t.Fatalf("NewPointFromBytes: got: (%v, %v), expected: (nil, error)", p, err)
models: add tests for point construction These tests check that NewPoint and NewPointFromBytes return an error if: * arguments specify a point with no fields * arguments specify a point with a field that has an empty name These tests also check that Point.Fields() always returns, even in the presence of corrupt binary data read with NewPointFromBytes. These tests fail at this commit and are fixed by a subsequent commit. Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-02-17 15:47:29 +00:00
}
}
Add point.UnmarshalBinary() bounds checking.
2017-03-01 18:43:13 +00:00
func TestNewPointsWithShortBuffer(t *testing.T) {
_, err := models.NewPointFromBytes([]byte{0, 0, 0, 3, 4})
if err != io.ErrShortBuffer {
t.Fatalf("NewPointFromBytes: got: (%v, %v), expected: (nil, error)", p, err)
}
}
models: add tests for point construction These tests check that NewPoint and NewPointFromBytes return an error if: * arguments specify a point with no fields * arguments specify a point with a field that has an empty name These tests also check that Point.Fields() always returns, even in the presence of corrupt binary data read with NewPointFromBytes. These tests fail at this commit and are fixed by a subsequent commit. Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2016-02-17 15:47:29 +00:00
func TestNewPointsRejectsEmptyFieldNames(t *testing.T) {
if _, err := models.NewPoint("foo", nil, models.Fields{"": 1}, time.Now()); err == nil {
t.Fatalf("new point with empty field name. got: nil, expected: error")
}
}
Return error if creating a point would exceed max key length
2016-03-31 05:57:41 +00:00
func TestNewPointsRejectsMaxKey(t *testing.T) {
var key string
Fix point validation to include field key length The series key stored in TSM files includes the field. We validated the series length using only the measurement and tag set which allowed very large field names to overflow. This now checks the series key as the measurement + tagset + field + the tsm field key separator size.
2017-05-24 20:35:34 +00:00
// tsm field key is point key, separator (4 bytes) and field
for i := 0; i < models.MaxKeyLength-len("value")-4; i++ {
Return error if creating a point would exceed max key length
2016-03-31 05:57:41 +00:00
key += "a"
}
Fix point validation to include field key length The series key stored in TSM files includes the field. We validated the series length using only the measurement and tag set which allowed very large field names to overflow. This now checks the series key as the measurement + tagset + field + the tsm field key separator size.
2017-05-24 20:35:34 +00:00
// Test max key len
if _, err := models.NewPoint(key, nil, models.Fields{"value": 1, "ok": 2.0}, time.Now()); err != nil {
t.Fatalf("new point with max key. got: %v, expected: nil", err)
}
if _, err := models.ParsePointsString(fmt.Sprintf("%v value=1,ok=2.0", key)); err != nil {
t.Fatalf("parse point with max key. got: %v, expected: nil", err)
}
// Test 1 byte over max key len
key += "a"
if _, err := models.NewPoint(key, nil, models.Fields{"value": 1, "ok": 2.0}, time.Now()); err == nil {
Return error if creating a point would exceed max key length
2016-03-31 05:57:41 +00:00
t.Fatalf("new point with max key. got: nil, expected: error")
}
Fix point validation to include field key length The series key stored in TSM files includes the field. We validated the series length using only the measurement and tag set which allowed very large field names to overflow. This now checks the series key as the measurement + tagset + field + the tsm field key separator size.
2017-05-24 20:35:34 +00:00
if _, err := models.ParsePointsString(fmt.Sprintf("%v value=1,ok=2.0", key)); err == nil {
Return error if creating a point would exceed max key length
2016-03-31 05:57:41 +00:00
t.Fatalf("parse point with max key. got: nil, expected: error")
}
Fix point validation to include field key length The series key stored in TSM files includes the field. We validated the series length using only the measurement and tag set which allowed very large field names to overflow. This now checks the series key as the measurement + tagset + field + the tsm field key separator size.
2017-05-24 20:35:34 +00:00
Return error if creating a point would exceed max key length
2016-03-31 05:57:41 +00:00
}
Fix panic with parsing empty key Fixes #6990
2016-07-28 00:01:29 +00:00
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
func TestPoint_FieldIterator_Simple(t *testing.T) {
p, err := models.ParsePoints([]byte(`m v=42i,f=42 36`))
if err != nil {
t.Fatal(err)
}
if len(p) != 1 {
t.Fatalf("wrong number of points, got %d, exp %d", len(p), 1)
}
fi := p[0].FieldIterator()
if !fi.Next() {
t.Fatal("field iterator terminated before first field")
}
if fi.Type() != models.Integer {
t.Fatalf("'42i' should be an Integer, got %v", fi.Type())
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
iv, err := fi.IntegerValue()
if err != nil {
t.Fatal(err)
}
if exp, got := int64(42), iv; exp != got {
t.Fatalf("'42i' should be %d, got %d", exp, got)
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
}
if !fi.Next() {
t.Fatalf("field iterator terminated before second field")
}
if fi.Type() != models.Float {
t.Fatalf("'42' should be a Float, got %v", fi.Type())
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
fv, err := fi.FloatValue()
if err != nil {
t.Fatal(err)
}
if exp, got := 42.0, fv; exp != got {
t.Fatalf("'42' should be %f, got %f", exp, got)
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
}
if fi.Next() {
t.Fatal("field iterator didn't terminate")
}
}
func toFields(fi models.FieldIterator) models.Fields {
m := make(models.Fields)
for fi.Next() {
var v interface{}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
var err error
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
switch fi.Type() {
case models.Float:
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
v, err = fi.FloatValue()
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
case models.Integer:
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
v, err = fi.IntegerValue()
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
case models.Unsigned:
v, err = fi.UnsignedValue()
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
case models.String:
v = fi.StringValue()
case models.Boolean:
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
v, err = fi.BooleanValue()
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
case models.Empty:
v = nil
default:
panic("unknown type")
}
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
if err != nil {
panic(err)
}
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
m[string(fi.FieldKey())] = v
}
return m
}
func TestPoint_FieldIterator_FieldMap(t *testing.T) {
points, err := models.ParsePointsString(`
m v=42
m v=42i
m v="string"
m v=true
m v="string\"with\"escapes"
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
m v=42i,f=42,g=42.314,u=123u
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
m a=2i,b=3i,c=true,d="stuff",e=-0.23,f=123.456
`)
if err != nil {
t.Fatal("failed to parse test points:", err)
}
for _, p := range points {
panicing is bad -> error returns are good
2017-01-03 17:54:17 +00:00
exp, err := p.Fields()
if err != nil {
t.Fatal(err)
}
models: Add FieldIterator type The FieldIterator is used to scan over the fields of a point, providing information, and delaying parsing/decoding the value until it is needed. This change uses this new type to avoid the allocation of a map for the fields which is then thrown away as soon as the points get converted into columns within the datastore.
2016-09-21 17:12:09 +00:00
got := toFields(p.FieldIterator())
if !reflect.DeepEqual(got, exp) {
t.Errorf("FieldIterator failed for %#q: got %#v, exp %#v", p.String(), got, exp)
}
}
}
Use strings.Replacer to escape string field benchmark old ns/op new ns/op delta BenchmarkEscapeStringField_Plain-4 167 65.3 -60.90% BenchmarkEscapeString_Quotes-4 167 165 -1.20% BenchmarkEscapeString_Backslashes-4 211 184 -12.80% BenchmarkEscapeString_QuotesAndBackslashes-4 413 397 -3.87% BenchmarkExportTSMStrings_100s_250vps-4 33833611 27381442 -19.07% BenchmarkExportWALStrings_100s_250vps-4 34977761 29222717 -16.45% benchmark old allocs new allocs delta BenchmarkEscapeStringField_Plain-4 4 1 -75.00% BenchmarkEscapeString_Quotes-4 4 3 -25.00% BenchmarkEscapeString_Backslashes-4 5 3 -40.00% BenchmarkEscapeString_QuotesAndBackslashes-4 9 5 -44.44% BenchmarkExportTSMStrings_100s_250vps-4 201605 76938 -61.84% BenchmarkExportWALStrings_100s_250vps-4 225371 100728 -55.31% benchmark old bytes new bytes delta BenchmarkEscapeStringField_Plain-4 56 16 -71.43% BenchmarkEscapeString_Quotes-4 56 48 -14.29% BenchmarkEscapeString_Backslashes-4 104 80 -23.08% BenchmarkEscapeString_QuotesAndBackslashes-4 208 160 -23.08% BenchmarkExportTSMStrings_100s_250vps-4 10872629 6062048 -44.24% BenchmarkExportWALStrings_100s_250vps-4 10094933 5269980 -47.80%
2016-12-18 07:37:25 +00:00
func TestEscapeStringField(t *testing.T) {
cases := []struct {
in string
expOut string
}{
{in: "abcdefg", expOut: "abcdefg"},
{in: `one double quote " .`, expOut: `one double quote \" .`},
{in: `quote " then backslash \ .`, expOut: `quote \" then backslash \\ .`},
{in: `backslash \ then quote " .`, expOut: `backslash \\ then quote \" .`},
}
for _, c := range cases {
// Unescapes as expected.
got := models.EscapeStringField(c.in)
if got != c.expOut {
t.Errorf("unexpected result from EscapeStringField(%s)\ngot [%s]\nexp [%s]\n", c.in, got, c.expOut)
continue
}
pointLine := fmt.Sprintf(`t s="%s"`, got)
test(t, pointLine, NewTestPoint(
"t",
models.NewTags(nil),
models.Fields{"s": c.in},
time.Unix(0, 0),
))
}
}
func BenchmarkEscapeStringField_Plain(b *testing.B) {
s := "nothing special"
for i := 0; i < b.N; i++ {
sink = models.EscapeStringField(s)
}
}
func BenchmarkEscapeString_Quotes(b *testing.B) {
s := `Hello, "world"`
for i := 0; i < b.N; i++ {
sink = models.EscapeStringField(s)
}
}
func BenchmarkEscapeString_Backslashes(b *testing.B) {
s := `C:\windows\system32`
for i := 0; i < b.N; i++ {
sink = models.EscapeStringField(s)
}
}
func BenchmarkEscapeString_QuotesAndBackslashes(b *testing.B) {
s1 := `a quote " then backslash \ .`
s2 := `a backslash \ then quote " .`
for i := 0; i < b.N; i++ {
sink = [...]string{models.EscapeStringField(s1), models.EscapeStringField(s2)}
}
}
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
inmem startup improvments * only call ParseTags when necessary * remove dependency on inmem.Series in tsdb test package * Measurement and Series are no longer exported. Their use is restricted to the inmem package * improve Measurement and Series types by exporting immutable fields and removing unnecessary APIs and locks Reduced startup time from 28s to 17s. Overall improvement including #9162 reduces startup from 46s to 17s for 1MM series across 14 shards.
2017-11-27 21:13:02 +00:00
func BenchmarkParseTags(b *testing.B) {
tags := []byte("cpu,tag0=value0,tag1=value1,tag2=value2,tag3=value3,tag4=value4,tag5=value5")
for i := 0; i < b.N; i++ {
models.ParseTags(tags)
}
}
Add uint support into the write protocol This is currently protected behind a conditional compilation flag. Use `-tags uint` or `-tags uint64` to enable this.
2017-09-14 19:01:07 +00:00
func init() {
// Force uint support to be enabled for testing.
models.EnableUintSupport()
}