09d7dfbaae
Form database path correctly on DROP DATABASE
...
Fixes #3330
2015-07-15 11:14:49 -07:00
ebd6e55997
Fix regression parsing boolean True/False values
2015-07-13 11:29:02 -06:00
f4f0373579
Sort points after appending if needed
...
Writing points that were not sorted by time could cause very high
CPU usages and increased latencies because each point inserted would
cause the in-memory cache to be resorted. The worst case would be
writing a large batch of N points in reverse time order which would
invoke N sorts of the slice.
This patch keeps track of which slices need to be sorted and sorts
them once at the end. In the previous example, the N sorts becomes
one. There is still a pathalogical case that would require N/2 sorts.
For example, 10000 points split across 5000 series. Each series has two
points that are in reverse time order. This would incur 5000 sorts still.
Fixes #3159
2015-07-13 10:51:58 -06:00
1641c25479
Fix panic parsing floats without decimal
...
Fixes #3289
2015-07-10 14:18:51 -06:00
7a87fbc58a
Flush WAL as fast as possible on start-up
...
This addresses complaints of long start-up times when there is lots of
data sitting in the WAL.
2015-07-08 17:14:28 -06:00
6b8d3268e6
Fix code review comments
2015-07-07 11:41:12 -06:00
cb75de8b7c
Use UTC for test assertion data
2015-07-06 21:11:09 -06:00
351bc03655
Don't panic if NaN or -/+Inf is used as a field value
...
These are not supported types but previously it would cause the
point.Fields() func to panic. This prevents it from panicing
so the values can be ignored if needed.
2015-07-06 16:14:02 -06:00
4d511571c5
Serialized unknown types as strings
...
When creating a point manually, the field values are interface{}
which allows unsupported types to be passed in. Previously, the
code would panic. It will now default to string representation of
the value if it's not a known type.
2015-07-06 16:14:02 -06:00
736f1c142e
Only start flush timer if duration is set
...
The default of 0 causes the time timer to flush immediately. If
only a batch size is set but not a time, the batch size setting does
not work.
2015-07-06 16:14:02 -06:00
a3ab093996
Parse NaN as float
...
Fixes #3230
2015-07-06 16:14:01 -06:00
dd66491f65
stringSet now returns elements in sorted order
2015-07-06 12:03:58 -04:00
ca86fa2633
Allow WAL inter-flush time to be configurable
2015-07-02 10:40:26 -04:00
39af8e49de
Remove obsolete retention config from tsdb
2015-07-01 09:39:11 -04:00
071c985b5b
Remove obsolete retention auto-create from tsdb
2015-07-01 09:37:29 -04:00
babc63d941
Use typed error for "field not found"
2015-06-30 14:17:04 -04:00
7a284885f2
Add helper to decode specific field by name
2015-06-30 14:16:54 -04:00
fde3d1f6a2
Remove unused, unexported function
2015-06-30 13:28:56 -04:00
41ae8bdae7
Handle escaped commas in measurement name
...
Fixes #3183
2015-06-29 15:15:50 -06:00
eb71f78afb
Merge pull request #3167 from influxdb/jw-2608
...
Fix panic when droppping measurement while writing to it concurrently
2015-06-29 14:05:42 -06:00
7232e6ea7c
Fix panic when droppping measurement while writing to it concurrently
...
Fixes #2608
2015-06-29 14:01:07 -06:00
5fccee3d16
Fix spelling errors in comments and strings.
2015-06-28 02:54:34 -04:00
1a5a8d1675
Fix typos in comments
2015-06-26 14:36:19 -06:00
d33b8bcd6a
Return parsing error when tag name is missing
...
Fixes #2678
2015-06-26 13:47:09 -06:00
5081481ca8
Return error when parsing fields with no name
...
Fixes #3061
2015-06-26 13:39:25 -06:00
b574e2f755
Add write ahead log
...
This commit adds a write ahead log to the shard. Entries are cached
in memory and periodically flushed back into the index. The WAL and
the cache are both partitioned into buckets so that flushing doesn't
stop the world as long.
2015-06-25 15:47:13 -06:00
17432598b1
Prevent out of range ints and floats from being parsed sucessfully
...
Field values that were out of range for the type would panic the database
when being inserted because the parser would allow them as valid points.
This change prevents those invalid values from being parsed and instead
returns an error.
An alternative fix considered was to handle the error and clamp the value
to the min/max value for the type. This would treat numeric range errors
slightly differently than other type erros which might lead to confusion.
The simplest fix with the current parser would be to just convert each field
to the type at parse time. Unfortunately, this adds extra memory allocations
and lowers throughput significantly. Since out of range values are less common
than in-range values, some heuristics are used to determine when the more
expensive type parsing and range checking is performed. Essentially, we only
do the slow path when we cannot determine that the value is in an acceptable
type range.
Fixes #3127
2015-06-25 14:49:42 -06:00
01440f90e9
Merge pull request #3095 from influxdb/beckettsean-patch-3
...
more examples for line protocol
2015-06-23 15:20:26 -07:00
c2a1511e2e
Update README.md
2015-06-23 15:19:53 -07:00
92b38583f3
more examples for line protocol
...
showing escape characters and proper string quoting
2015-06-22 17:10:24 -07:00
1d9aa31e7b
Merge pull request #3072 from influxdb/beckettsean-patch-3
...
typos, clarifications in line protocol doc
2015-06-22 17:00:31 -07:00
cb7baa6d9e
Don't group TagSets when tag values are identical
...
Fixes issue #3059
2015-06-22 16:04:13 -07:00
7873ccbb20
Merge pull request #3039 from neonstalwart/line-comments
...
allow comments in line protocol
2015-06-22 14:09:56 -06:00
cc7e59a12a
Merge pull request #3088 from influxdb/jw-field-values
...
Fix string field value escaping
2015-06-22 13:48:47 -06:00
2854108941
Fix string field value escaping
...
Commas and quotes could get escaped and parsed incorrectly if they
were both present in a string value.
Fixes #3013
2015-06-22 13:17:35 -06:00
01700f79b0
On DROP DATABASE remove from TSDB index
2015-06-22 11:44:46 -07:00
cb9a40df64
Fix invalid field value format not parsed correctly
...
A field value of just a numeric value would be accepted by the line
protocol parser but the value would be set as the field name and
the value would be nil. Instead, return an error because all field
values need a field name.
2015-06-22 10:35:08 -06:00
6911e752af
typos, clarifications
2015-06-19 17:58:28 -07:00
72a76e73f3
Add full example of line protocol
2015-06-19 08:14:38 -07:00
d6eeb12d67
Merge pull request #2997 from influxdb/jw-line-protocol
...
Add line protocol doc
2015-06-19 08:09:12 -07:00
b6252868cc
Add support for marshaling `uint64` in client.
2015-06-18 15:33:37 -07:00
d4a341a5ae
allow comments in line protocol
2015-06-18 14:03:27 -05:00
17be1e2f67
Remove measurement from shard index on a DROP
...
Fixes issue #2955
2015-06-16 12:14:33 -07:00
e0bfd827dc
Add line protocol doc
2015-06-15 16:47:04 -06:00
014eee6fa1
Remove stats support
...
This will soon be re-implemented using the expvar package.
2015-06-15 13:33:29 -07:00
05bd0fc53d
Check database existence during normalization
...
Statements were only being normalized if a default database was included
in the query (usually via the query param 'db'). However if no default
database was included, and none was an explicit part of the measurement
name, no database-existence check was run. This result in a later panic
with wildcard expansion.
2015-06-15 11:51:32 -07:00
e60645e1a2
Remove unused database param
...
It is not used by the code -- the database on the measurement object
that determines the target database.
2015-06-15 11:09:16 -07:00
36cde5f35e
Fix large integers getting converted to floats during remote writes
...
Fixes #2960
Integers were were written back to line protocol using strconv.FormatFloat
incorrectly. Large integers are written in scientific notation which
causes their type to change to a float when parsed back.
2015-06-12 11:20:37 -06:00
30068561ae
Fix parsing negative floats
...
Fixes #2919
2015-06-12 08:39:17 -06:00
861f63e33e
Merge pull request #2949 from influxdb/jw-error-responses
...
Add measurement name to type conflict error messages
2015-06-11 19:59:55 -06:00
be5491ea77
Merge pull request #2918 from pradeepchhetri/master
...
Fixed some typos
2015-06-11 18:28:24 -07:00
20fe5b0218
Add measurement name to type conflict error messages
...
Fixes #2948
2015-06-11 16:16:07 -06:00
c97dedfa3d
Be more explicit about boolean value parsing
...
Supported boolean values are now t, T, true, TRUE, f, F, false, and
FALSE. This is what the strconv.ParseBool function supports with
the exception of 1 and 0. 1 and 0 would be parsed as ints in the
line protocol.
Previously, any non-true value would be parsed as false. e.g.
value=blah would parse to false. This will now return an error as
parsing time.
2015-06-11 14:53:53 -06:00
6d115a7552
Return the line that failed to parse
...
Make it easier to find a bad line in a batch when writing points.
2015-06-11 13:59:30 -06:00
0d82acfde7
Prevent invalid numbers from being parsed via line protocol
...
Adds more tests for invalid numbers such as 0.1a, -2.-4, as well
test for supported formats for negative and positive integers/floats
as well as scientific notation.
2015-06-11 13:50:28 -06:00
b497256dc7
Fix panic when no value is passed to a field
...
Fixes #2927
2015-06-11 11:25:25 -06:00
37750acef6
Fixed some Typos
2015-06-11 17:33:26 +05:45
64af1b6241
Report number of measurements and series per node
2015-06-11 00:21:15 -07:00
ee5406d8a0
Merge pull request #2901 from influxdb/no_query_panic
...
SHOW DIAGNOSTICS is not yet (re)implemented
2015-06-10 21:12:02 -07:00
ffda846f33
SHOW DIAGNOSTICS is not yet (re)implemented
2015-06-10 21:09:25 -07:00
bc31783a00
Refactor backup and restore
...
This commit updates the snapshot code as well as the "backup" and
"restore" command to work with the new architecture.
2015-06-10 22:07:01 -06:00
325ca4ab4b
Merge pull request #2885 from influxdb/datatype-refactor
...
change influxql DataType from string to int
2015-06-10 21:06:08 -07:00
6d26f9c8a8
Merge pull request #2856 from influxdb/rv_fixes
...
Series was not already dropped, return false (and other fixes)
2015-06-10 20:57:46 -07:00
344a1f4948
Don't even return value from DropSeries
2015-06-10 20:50:07 -07:00
5ead14c1d0
Unit test returning empty set when no database
2015-06-10 19:22:16 -07:00
4c2c1ae1dd
Return empty results set if no shard meta database
2015-06-10 19:22:16 -07:00
10ca99ffb8
SHOW DIAGNOSTICS is not yet (re)implemented
2015-06-10 15:49:00 -07:00
0d9a2a5053
Fix wrong value used for incorrect type error message
...
Would result in saying that the field type was always a string which
is incorrect. The field key is a string but the value may not be.
Fixes #2700
2015-06-10 15:57:27 -06:00
67d4ef0e28
Don't queue write failures that due to type conflicts
...
These will never succeed and will stay in the queue indefinitely.
2015-06-10 14:52:59 -06:00
999f4a4c41
Return field type errors as client write errors
...
Fixes #2849
2015-06-10 14:52:26 -06:00
29777f2a8e
change influxql DataType from string to int
2015-06-10 16:02:26 -04:00
bc7e1f6fd6
Fix panic when adding new fields
...
Fixes #2869
When adding a new field to an existing measurment, Shard.validateSeriesAndFields
would also encode the fields as a side-effect. In the case of a new field
that needed to be created, the encoding would fail because the field type
had not been created for the measurement yet. The fields are re-encoded
after validateSeriesAndFields returns and after the field encoding have been
setup properly so this additional encoding during
validation isn't necessary.
2015-06-10 10:30:14 -06:00
98d315b8d8
Ensure tags are always marshalled the same way
...
This added check ensures it's always alphabetically ordered when
marshalled.
2015-06-09 17:10:46 -07:00
8a5b337ae5
There is no more metastore, so correct comment
2015-06-09 14:57:19 -07:00
3e969166c9
Fix parsing commas in string field values
...
Fixes a panic on writes because the field value was not parse correctly.
panic: unsupported value type during encode fields: <nil>
goroutine 117 [running]:
github.com/influxdb/influxdb/tsdb.(*FieldCodec).EncodeFields(0xc2081c4020, 0xc2081dc180, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/jason/go/src/github.com/influxdb/influxdb/tsdb/shard.go:573 +0x8e3
2015-06-09 15:53:21 -06:00
bc68afbe85
Fix minor spelling mistake
2015-06-09 14:35:38 -07:00
85fd3d0292
Series was not already dropped, return false
2015-06-09 14:25:20 -07:00
a43054adce
add test for querying across shards and fields
2015-06-09 12:51:59 -05:00
fc0bbaf0d1
fix #2814 : remove locking cruft from batcher
2015-06-09 00:34:46 -04:00
3b15a49f7c
fix #2814 : revert making batcher channels buffered
2015-06-09 00:34:46 -04:00
d5f52333a1
fix #2814 : hook collectd service back up
2015-06-09 00:34:46 -04:00
8f712eaf85
Ignore shards that don't have the selected measurement in them.
...
Fixes #2815 and fixes #2818 .
2015-06-08 17:14:42 +02:00
f39cddebf6
Fix locking on write path for getting measurement field encoding.
2015-06-07 11:09:47 +02:00
044e5b71f5
Remote temp dirs in store_test
2015-06-05 22:16:51 -06:00
b925e1c1af
Multi-node clustering.
...
This commit adds the ability to cluster multiple nodes together to share
the same metadata through raft consensus.
2015-06-05 14:41:19 -06:00
0ee71b9755
Merge pull request #2743 from influxdb/tsdb-benchmarks
...
add shard & index benchmarks
2015-06-05 13:15:38 -07:00
ca9f231fa1
Merge pull request #2776 from influxdb/enforce_retention_2_phase
...
Enforce retention policies
2015-06-05 11:15:21 -07:00
4ba7c5d1c1
Merge pull request #2787 from influxdb/pd-drop-database
...
Wire up DROP DATABASE query
2015-06-05 20:09:06 +02:00
61ac4aad80
Fix spelling in test
2015-06-05 20:07:52 +02:00
885e484746
Fix comment typo
2015-06-05 12:47:53 -04:00
70e10b136e
Wire up DROP DATABASE query.
2015-06-05 12:31:04 -04:00
81ec7353ce
TSDB store can return all known Shard IDs
2015-06-04 17:02:22 -07:00
55e0de30bd
Merge pull request #2773 from influxdb/pd-locking
...
Ensure proper locking of index structures on writes and queries.
2015-06-04 19:50:49 -04:00
9bf09ee026
Correct comments in tsdb/meta
2015-06-04 16:08:12 -04:00
c36f5f3f5f
Merge pull request #2771 from influxdb/enforce_retention_2_phase
...
Support deleting shards from TSDB store
2015-06-04 12:32:39 -07:00
5ea2042356
Add DeleteShard to TSDB store
2015-06-04 12:22:38 -07:00
408bc3f81e
Ensure proper locking of index structures on writes and queries.
2015-06-04 14:50:32 -04:00
bafdb7f7cb
explicitly parse time if needed
2015-06-04 12:15:18 -06:00
780fdedbd4
remove spew
2015-06-04 11:29:18 -06:00
47d605b69a
make line protocol round or parse precision as expected
2015-06-04 11:22:53 -06:00
3b9738c87a
fixing more error message regressions
2015-06-03 15:36:18 -06:00
3e7b458eae
fix regression in distinct on tags error
2015-06-03 15:28:45 -06:00
99446786f9
Merge pull request #2752 from influxdb/pd-drop-measurement
...
Wire up DROP MEASUREMENT
2015-06-03 14:02:37 -04:00
f5d59eca3d
Merge pull request #2756 from influxdb/integration-tests
...
And even more Integration tests
2015-06-03 11:56:36 -06:00
be91c40b9a
fix data race in WriteToShard
2015-06-03 11:46:18 -06:00
a7685767f8
Update comment
2015-06-03 13:26:49 -04:00
73a6c7e424
Merge pull request #2745 from influxdb/pd-shard-metadata-to-protobuf
...
Update metadata storage in the shard to use protobuf for serialization.
2015-06-03 11:33:25 -04:00
9c4da3002c
Wire up DROP MEASUREMENT
...
* Add deleteMeasurement to store and shard
* Add DropMeasurement to DatabaseIndex
* Update ErrMeasurementNotFound and ErrDatabaseNotFound to not include the first line of the stack trace.
2015-06-03 11:32:50 -04:00
938ad2ef85
add Store Open benchmark test
2015-06-03 10:09:50 -04:00
808e50f1cb
Add protbuf definitions and code
2015-06-03 09:16:38 -04:00
6f76a4774b
Update UnmarshalBinary comment to be correct
2015-06-03 07:36:39 -04:00
2d9c63f3bd
Update metadata storage in the shard to use protobuf for serialization.
2015-06-02 19:08:48 -04:00
31bb8e70a9
don't build index before benchmarking WritePoints
2015-06-02 17:17:31 -04:00
97c84a6d4f
add benchmark tests for shard WritePoints
2015-06-02 17:00:25 -04:00
67464238ed
add benchmarks for building in-mem series index
2015-06-02 17:00:25 -04:00
900903ee69
fix select tags
2015-06-02 14:48:33 -06:00
b18546be22
Fix tx.go after merge of alpha1
2015-06-02 14:56:20 -04:00
44d38cb430
Merge branch 'master' into alpha1
2015-06-02 10:40:52 -06:00
c098c6fd78
Merge pull request #2728 from influxdb/pd-tsdb-drop-queries
...
Wire up DROP SERIES.
2015-06-02 11:30:49 -04:00
4a1ce77090
Wire up DROP SERIES.
...
* Pulled over updates to ast and parser from master
* Updated store and shard to be able to drop series
* Pulled updates to database.go from master into tsdb/meta.go
2015-06-02 11:20:20 -04:00
4fa9fd5fe9
Merge pull request #2725 from influxdb/jw-client
...
Update client to use line protocol
2015-06-01 17:35:26 -06:00
5c851c4b1c
Fix panic: runtime error: slice bounds out of range in tsdb.measurementFromSeriesKey
...
Writing points without tags caused this since strings.Index(key, ",") returned
-1 if there were no tags.
2015-06-01 17:19:29 -06:00
d962283ae6
adding some integration tests
2015-06-01 16:04:20 -06:00
3597565955
reading and writing yo!
2015-06-01 11:59:58 -06:00
bf823d9887
Integrating cmd/influxd/run.
2015-05-30 14:06:36 -06:00
c916256ac9
Rename cluster.Writer to cluster.ShardWriter.
2015-05-30 14:05:27 -06:00
f3245ab9ac
Make Authenticate return true if no users and creating a root user.
2015-05-30 14:29:16 -04:00
89aee31a5d
Add comments and stub out `executeDropDatabase`
2015-05-30 12:10:24 -04:00
15d37fd388
Make store open every shard on load. Fix shard to set measurementFields on load.
...
Fixes issue where queries wouldn't be able to hit anything because the index does't load until the shard is open.
Fix an issue where field codecs weren't populated in the shard when loading.
2015-05-30 11:53:53 -04:00
cdc5a47efa
Clean up influxdb.
2015-05-30 08:14:10 -06:00
97f9670fa3
Fix influxql.Rows.
2015-05-30 07:19:19 -06:00
f66afc23bb
merging alpha1, fix vet errors
2015-05-29 15:28:08 -06:00
9d4527071e
Refactor run command.
2015-05-29 14:59:57 -06:00
1f294ce8de
Add httpd.Handler.serveQuery() tests.
2015-05-29 14:59:57 -06:00
f660b1a3d3
Create QueryExecutor in tsdb.
2015-05-29 14:59:57 -06:00
072dacc1ab
Update tx.go to work with tsdb
2015-05-29 14:59:57 -06:00
df1aeee70a
WIP
2015-05-29 14:56:30 -06:00
75ab63b538
Refactor meta statement execution to meta.StatementExecutor.
2015-05-29 14:54:04 -06:00
870a183576
Ensure comma,space and equals are escaped
2015-05-29 14:34:06 -06:00
4e7c8bdad2
Add ParsePointsWithPrecision to handle precision write argument
2015-05-29 11:18:40 -06:00
e1322bb7d9
Add initial write_points http handler for text protocol
2015-05-29 11:18:40 -06:00
9a9bb736f7
Add text protocol parsing and serialzation for points
...
This changes the implementation of point to minimize the extra
processing needed to parse and marshal point data though the system.
2015-05-29 11:18:40 -06:00
85f59d696b
Create and open shards on-demand
...
Uses a structure like:
/root/
/db1/rp1/1
/2
/db2/rp2/3
If a write is assigned to a shard on the local node but the shard
has not been created, create it when the write returns an error
and retry the write.
2015-05-26 16:38:45 -06:00
7ace10f7e6
Load shards from filesystem when tsdb.Store is opened
2015-05-26 15:43:46 -06:00
01618dc143
Move data.Node to tsdb.Store. Move data to cluster.
2015-05-26 15:56:54 -04:00
21bfb150a1
Wire up new write path
...
This allows the new write path to be hooked up if you start the
server with `INFLUXDB_ALPHA1=1`. When set, writes will go though
the coordinator and be stubbed out to write to a single local data
node with one shards. The write will be logged and written to
disk .
The env var is used so that the current write path is not completely
broken which would break many of the tests that depend on writes.
Note that queries are not currently working w/ the this change.
2015-05-26 12:07:56 -06:00
6c80108f63
Change Database to DatabaseIndex, remove leftover warn statement
2015-05-24 07:39:45 -04:00
c3ab88a715
Make the metadata index shared across shards while keeping field types and encoding local to each shard.
2015-05-23 18:06:07 -04:00
1076153a00
Convert Point to interface
...
Should be possible to replace the implementation with a more
optimized version now.
2015-05-22 15:39:55 -06:00
528f47e093
Convert Point.Fields to Point.Fields()
2015-05-22 15:22:03 -06:00
f8d599cda9
Convert Point.Tags to Point.Tags()
2015-05-22 15:12:34 -06:00
5dcab443dc
Move data.Point to tsdb.Point
2015-05-22 15:00:51 -06:00
8f937cae87
Initial implementation for writing data to a shard.
2015-05-22 16:11:18 -04:00
Philip O'Toole
Jason Wilder
Philip O'Toole
Joseph Crail
Ben Johnson
Sean Beckett
Todd Persen
ben hockey
Pradeep Chhetri
Paul Dix
David Norton
Cory LaNou
Todd Persen