5b0767c30b
EOF is OK in HH processor
2015-10-07 01:56:55 -07:00
8b49c37120
Count HH errors
2015-10-06 20:49:40 -07:00
5d5515a497
If HH can't unmarshal a block, skip that block
2015-10-06 20:49:40 -07:00
883d32cfd0
Add public function to graphite parser to apply template
2015-10-06 17:42:36 -06:00
bb398daf75
Updates based on @otoolp's PR comments
2015-10-05 20:09:56 -04:00
5d9b89d601
Disable copier test
...
Not implemented for tsm1 engine
2015-10-05 20:09:56 -04:00
7555ccbd70
WIP: engine work
2015-10-05 20:06:21 -04:00
2ac0357406
Support dropping non-Raft nodes
2015-10-04 00:19:52 -07:00
d74e0690c7
Revert "Merge pull request #4233 from influxdb/drop-server"
...
This reverts commit 0bdb36f6dc3085fbc138
2015-10-02 08:39:57 -07:00
f50813460e
protobuf update.. :-(
2015-10-01 15:39:15 -05:00
8a1e5a9e53
Clamp initial value of HH retry interval
...
This could happen due to misconfiguration, so do something sensible in
that case.
2015-10-01 12:04:33 -07:00
878f776403
Exponential backoff if any hinted-handoff fails
2015-09-30 21:27:13 -07:00
4eba2c1725
Add config support for max HH retry interval
2015-09-30 21:10:03 -07:00
235714755c
HH processor-level stats
...
This change maintains stats on a per-shard and per-node basis.
2015-09-28 18:39:39 -07:00
14db3ce9f5
Add service-level stats for hinted-handoff
2015-09-28 18:08:35 -07:00
a196d3663a
Allow configuration of UDP retention policy
...
Fixes issue #4529
2015-09-28 15:17:56 -07:00
49a70d0fca
Merge pull request #4238 from influxdb/hh_control
...
Fully disable hinted-handoff service if requested
2015-09-28 12:11:18 -07:00
a4a8fa0ff0
Fully disable hinted-handoff service if requested
...
Without this change if hinted-handoff was disabled the service would
correctly reject writes, but it would process any data sitting in
hinted-handoff queues. With this change the service is completely
disabled.
2015-09-25 18:03:43 -07:00
9de3125f6b
Graphite TCP should not block system shutdown
...
With this change Graphite TCP connections are tracked on a per-service
basis. This allows a closing Graphite service to first shutdown any
active connections, thereby unblocking the rest of shutdowm.
This work exposed small shortcomings with the existing Diagnostics
system and that code has alse been tweaked.
Fixes issue #4017
2015-09-24 14:08:38 -07:00
49c0b6ea73
Fix go vet warnings
...
This patch fixes the following go vet warnings:
```
services/continuous_querier/service.go:326: influxql.Statements
composite literal uses unkeyed fields
exit status 1
services/httpd/handler_test.go:145: models.Rows composite literal uses
unkeyed fields
services/httpd/handler_test.go:146: models.Rows composite literal uses
unkeyed fields
services/httpd/handler_test.go:165: models.Rows composite literal uses
unkeyed fields
services/httpd/handler_test.go:166: models.Rows composite literal uses
unkeyed fields
services/httpd/handler_test.go:187: models.Rows composite literal uses
unkeyed fields
services/httpd/handler_test.go:188: models.Rows composite literal uses
unkeyed fields
exit status 1
```
Signed-off-by: Antonio Murdaca <runcom@linux.com>
2015-09-21 15:28:54 +02:00
ef35d6dcc2
formatting
2015-09-21 12:26:43 +01:00
27c1cc23fd
Working prototype..
2015-09-21 12:18:19 +01:00
9121b422f8
comment out tests for now..
2015-09-21 10:47:42 +01:00
20e4fdfa9a
allow specifying fieldname in graphite template
2015-09-20 21:17:50 +01:00
72f6f7d268
Merge pull request #4134 from influxdb/issue-3447
...
Refactor Points and Rows to dedicated packages
2015-09-17 15:27:48 -05:00
38cb7b49de
Mising defer in httpd recovery. fixes #4124
2015-09-17 09:37:27 -05:00
ba830be3b9
actually move influxql.Row* -> models.Row*
2015-09-16 16:32:50 -05:00
d19a510ad2
refactor Points and Rows to dedicated packages
2015-09-16 15:33:08 -05:00
d538829b4c
Enhance openTSDB logging and stats
2015-09-09 13:30:11 -07:00
fef20c77b2
Cleanly terminate openTSDB connection on EOF
...
This is not really an error, so don't log it.
2015-09-09 13:01:13 -07:00
02fcaf853d
Add node re Graphite configuration
...
[ci skip]
2015-09-08 23:22:34 -07:00
519a30a463
Add note on openTSDB batching
...
[ci skip]
2015-09-08 23:19:17 -07:00
24aca5611a
Add batch-pending control to openTSDB input
2015-09-08 19:35:42 -07:00
95530e1623
Set UDP input defaults if not set
2015-09-08 19:32:20 -07:00
5373f263a3
Add pending control to batcher
...
With this change, the generic batcher used by many inputs can now be
buffered. Testing shows that this performance of the Graphite input by
10-100%, with the biggest improvements at lower numbers of connections.
2015-09-08 19:32:00 -07:00
e38a204afc
Merge pull request #4043 from influxdb/opentsdb_batching
...
Add batching and stats to openTSDB input
2015-09-08 19:27:35 -07:00
1ce5187b66
Merge pull request #4049 from influxdb/udp_stats
...
Add stats to the UDP input
2015-09-08 19:18:17 -07:00
9677a0faab
Add collectd stats
2015-09-08 19:07:47 -07:00
27932409b0
Add stats to the UDP input
2015-09-08 18:48:35 -07:00
817328d378
Add basic stats to the CQ service
2015-09-08 18:17:20 -07:00
349ba8b307
Add batching and stats to openTSDB input
2015-09-08 16:19:50 -07:00
73510a0a68
Fix invalid time stamp in graphite metric causes panic
...
If a timestamp was larger than the max epoch value was sent via
graphite it would cause the timestamp to overflow when it was
marshaled/unmarshaled back from the raft log. The overflow cause
the shard group to get created with the wrong timestamp which cause
a panic when writing the point. The panic was caused because the
timestamp that were supposed to exists in a map created by MapShards
did not actually exist so a nil ShardGroup was used.
The change prevents creating the point with an invalid timestamp. Since
graphite using a timestamp in seconds, the maximum range is known and
can be prevented. This also adds a check for the minimum range as well.
Fixes #3785
2015-09-08 10:07:47 -06:00
332ce6481d
Removed unused Graphite NewConfig
...
This function is not helpful for sections of the config that support
multiple instances.
2015-09-08 08:32:19 -07:00
bbc103305b
Support multiple Graphite inputs
...
Fixes issue #3636
2015-09-06 21:33:46 -07:00
fa29e12222
Shutdown UDP Graphite on SIGTERM
...
Service.Close() had no way of closing the UDP Conn. This change makes
the UDP an attribute of the server, so Close() can access it.
2015-09-05 00:30:59 -07:00
579e2a250c
Add stats to httpd package
2015-09-04 12:37:59 -07:00
3df898bd90
Merge pull request #3987 from influxdb/global_expvar_hookup_diagnostics
...
Use expvar statistics directly
2015-09-04 11:13:17 -07:00
89bc392ec4
Access expvar directly from monitor
...
expvar map is already global so access it directly. This simplifies the
code and makes it much eaisier to use from other modules.
2015-09-04 09:45:24 -07:00
cf5a655249
Don't precreate shard groups entirely in past
...
Fixes issue #3722
2015-09-04 08:31:50 -07:00
6ad35e23e9
Integrate code review feedback
2015-09-03 20:50:54 -07:00
d58532d844
Add Graphite diagnostics
...
Graphite diagnostics currently show TCP connections.
2015-09-03 20:50:54 -07:00
e07432c59f
Implement diagnostics support
...
This change adds support for diagnostics by decomposing the existing
interface into two interfaces -- one for stats, and the other for
diags. It also adds some basic monitor of system, network, and the Go
runtime.
2015-09-03 20:50:54 -07:00
dce666e757
fix #3979 : fix race in CQ service
2015-09-03 19:55:40 -04:00
deff06f850
add copier service
...
This commit adds the copier service which allows one server to
copy shards from another server. This will be used for moving
shards in the cluster.
2015-09-03 13:07:35 -06:00
0cb9618d6d
fix CQ intoDB()
2015-09-03 09:07:57 -04:00
d466b19388
update CQ service unit tests
2015-09-03 07:12:15 -04:00
66001cfbb5
fix #2555 : add integration tests for CQs
2015-09-03 07:12:15 -04:00
021a6f5453
rename CQ tests
2015-09-03 07:12:15 -04:00
99a22c174b
fix #2555 : add backreference in CQs
...
Add new query syntax to allow the following in CQs:
INTO "1hPolicy".:MEASUREMENT
2015-09-03 07:12:15 -04:00
4e2ee1ea70
Rename MonitorService to just Monitor
...
monitor is not a service, it has more in common with meta, since it
provides functionality to the query layer. This names makes this
clearer.
2015-09-02 15:07:30 -07:00
366c0115f9
Serve expvar information from HTTP package
2015-09-01 15:22:37 -07:00
9df17409d3
Use monitor service with Graphite
2015-09-01 15:21:36 -07:00
d87e668c78
Remove obsolete monitoring code
2015-09-01 15:03:52 -07:00
d771612718
Set default retention check interval to 30 minutes
...
Since the minimum retention period is 1 hour, checking every 10 minutes
seems excessive and generates noise in the logs.
2015-08-27 16:08:03 -07:00
ae825fdf3d
Correct typo in retention service logs
2015-08-27 16:08:03 -07:00
74dad8c68c
fix collectd tests for float data
2015-08-25 09:14:38 -05:00
6193226ce8
Revert "Merge pull request #3771 from influxdb/tcp_graphite_timeout"
...
This reverts commit d7f646f7a4d6f9903f10#3809
2015-08-24 10:53:14 -07:00
d7f646f7a4
Merge pull request #3771 from influxdb/tcp_graphite_timeout
...
Close idle Graphite TCP connections
2015-08-20 17:08:17 -07:00
50b0f67290
Add Graphite TCP timeout tests
2015-08-20 15:46:08 -07:00
4930a6d8bb
Start adding timeouts to TCP Graphite input
2015-08-20 15:10:22 -07:00
afe1f598ca
Cache name and fields if requested
...
Through profiling of writes, point.Fields() and point.Name() were called
repeatedly in PointsWriter and the Shard. These calls are somewhat expensive
when writing large batches so we can cache them to avoid wasting CPU cycles.
Using influx_stress with default settings
Before:
Wrote 10000000 points at average rate of 202570
Average response time: 235.450355ms
After:
Wrote 10000000 points at average rate of 246120
Average response time: 182.881008ms
2015-08-20 15:48:38 -06:00
8e51064db1
Log Graphite batch size and timeout
2015-08-20 11:23:09 -07:00
cf5ac2603d
Fix Graphite README typo
...
Fixes #3727
2015-08-19 07:53:29 -07:00
409fe0afe3
Merge pull request #3686 from jonseymour/secure-options
...
Prevent 'p' parameter of OPTIONS requests being logged.
2015-08-18 17:19:39 -07:00
5bb699e9a9
Enhance precreation log messages
2015-08-18 16:20:55 -07:00
bdce79fe57
Merge branch 'secure-options-minimal' into secure-options
2015-08-19 09:15:58 +10:00
1d5ff55d76
Remove redaction logic from parseCredentials.
...
We now redact the credentials in the logger, so the function implemented
by the deleted lines now seems redudndant.
Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2015-08-19 09:08:54 +10:00
28a6b1f3fd
Merge pull request #3697 from influxdb/chunking_10k
...
Merge same-series data if not chunking
2015-08-18 13:23:10 -07:00
2805c4a9b5
Ensure 'p' parameter is not logged, even on OPTIONS requests.
...
Previously password redaction only occurred inside the
authentication handler and the authentication handler is not on
the request path for OPTIONS requests and, in any case, would
not be invoked because of an early return on OPTIONS
requests by the CORS handler.
Now, we change the response logger to explictly replace any
occurrence of the 'p' parameter from the query string with
'[REDACTED]' prior to logging the response.
Signed-off-by: Jon Seymour <jon@wildducktheories.com>
2015-08-18 09:41:16 +10:00
6415944d01
Don't repeat retention policy log message
2015-08-17 16:15:51 -07:00
487c336571
Correctly merge rows for identical series
...
If no chunking was requested by the user, the co-ordinating node buffers all
results in RAM before emitting a single result. However buffering was not
merging results for rows which had data for the same series. This change fixes this.
Fixes issue #3242 .
2015-08-17 13:43:17 -07:00
7cf31a74cd
Prevent out of memory range slices from being created
...
If the hinted handoff segment is corrupt, the size read could be
invalid and attempting to create a slice using that size causes
a panic. Ideally, we'd have a checksum on the seqment record but
for now just return an error when the size is larger than the
segment file.
Fixes #3687
2015-08-17 10:48:01 -06:00
e5e782d13d
Merge pull request #3517 from dim/fix-cq-timeouts
...
Batch CQ writes to avoid timeouts
2015-08-14 10:52:17 -06:00
bb6de3b8f3
Merge pull request #3522 from dim/fix-cq-timeout-bug
...
Consume CQ results on request timeouts
2015-08-14 10:52:04 -06:00
0b05980ae2
silence snapshotter logger for testing
2015-08-13 20:53:40 -05:00
a7cb0df4af
Fix typos/spacing
2015-08-13 10:02:05 -06:00
668181d275
Make log statements more consistent
...
* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
2015-08-13 10:01:42 -06:00
089d947bf3
Shutdown Graphite listener first during Close()
...
Without this the WaitGroup was not fully decremented as the Accept()
call on the listener never exited, and Wait() then never exited.
2015-08-12 12:49:58 -07:00
966dee7559
Set sensible Graphite batching defaults
2015-08-11 18:34:36 -07:00
7dc7389e96
Remove dump from client and handler
2015-08-07 11:56:30 -07:00
398ffabab7
Fix panic in hinted handoff processor
...
A short write has occurred and we do not have enough bytes to determine
the size of the payload. This is corrupted record that we should drop.
Instead of panicing, log the error and advance the queue since the error
at this location is unreoverable currently.
Fixes #3436
2015-08-06 14:06:41 -06:00
4f7df336f2
Fix go vet
2015-08-05 12:16:17 -06:00
2fac5471ce
add TLS support to the OpenTSDB plugin
2015-07-31 12:00:22 -06:00
762a1c69d5
Consume CQ results on request timeouts
2015-07-30 16:59:47 +01:00
642d6eba85
Batch CQ writes to avoid timeouts
2015-07-30 15:24:51 +01:00
2a02605ef6
Merge pull request #3474 from influxdb/ga-options
...
Respond to OPTIONS requests on /query endpoint
2015-07-28 16:32:26 -07:00
fb76c34a79
Merge pull request #3426 from jhorwit2/jah/continuous-logging
...
Added additional logging to continuous queries
2015-07-27 17:43:55 -04:00
5ef0be2d71
Respond to OPTIONS requests on /query endpoint
2015-07-27 11:58:44 -07:00
38387ba6b7
Fixed typo.
2015-07-24 19:41:46 -04:00
5b50002728
Fixed typo in README
2015-07-24 17:07:41 -04:00
e722b4b4ad
Added additional logging to continuous queries
2015-07-23 19:09:42 -04:00
d9f16987fc
Merge pull request #3439 from influxdb/ga-admin-https
...
Add HTTPS option and logger to admin service
2015-07-23 15:09:56 -07:00
d1fc0a3cc9
Merge pull request #3375 from influxdb/https
...
First pass at re-enabling HTTPS.
2015-07-23 15:04:29 -07:00
b30351f750
Remove redundant loggers and clean up logic
2015-07-23 15:01:48 -07:00
8ab9424295
Return error if HTTPS fails
2015-07-23 15:00:33 -07:00
614332bf17
Exit if HTTPS fails
2015-07-23 14:50:45 -07:00
96575e678a
Merge pull request #3427 from influxdb/ga-pw-log
...
Logging tweaks, sanitize passwords and note if authentication is enabled
2015-07-23 14:12:41 -07:00
e5ead383e5
Add HTTPS option and logger to admin service
2015-07-22 16:49:12 -07:00
17cc09259b
change for better code clarity
2015-07-21 19:40:43 -06:00
785a8b4d9a
Sanitize password from HTTP logs
2015-07-21 18:28:05 -07:00
0780cf6599
Add a config test for HTTPS.
2015-07-21 18:20:08 -07:00
20de2bc914
Log authentication enabled message
2015-07-21 17:53:12 -07:00
3e29d8821a
catch opentsdb malformed tags if they are missing keys or values
2015-07-21 16:11:58 -06:00
425a65fca1
RemoteShard mapping now performed over TCP
...
With this change remote mapping no longer uses HTTP, as the HTTP ports
exposed by nodes on the cluster are not known cluster wide. The TCP
ports exposed by the cluster service are, so this change uses that
functionality. Each RemoteMapper has its own dedicated connection pool
for each node, and remote mapping TCP connections are in no way coupled
with query TCP connections.
2015-07-20 10:44:38 -07:00
47d5c2d65f
First pass at re-enabling HTTPS.
2015-07-17 16:57:31 -07:00
f549910a18
Merge pull request #3279 from LK4D4/fix_style_else
...
Fix style issues with else
2015-07-17 11:53:42 -07:00
9ba37325f6
Fixes authorization.
...
Adds GRANT and REVOKE statements for admin privilege. Adds authorization to the query endpoint.
2015-07-17 11:33:06 -07:00
675eacbf2c
Fix style issues with else
...
In go it's better to just continue flow without "else", if it is return in
"if" statement.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-07-17 11:10:23 -07:00
a0dba108a0
Unit test shard mapper handler error handling
2015-07-15 16:08:26 -07:00
65a580779b
Shard mapping handler returns on errors
2015-07-15 16:08:10 -07:00
7bdba556a1
Support pretty printing shard mapping
2015-07-15 13:06:16 -07:00
74cb96646c
Refactor query engine for distributed query support
...
With this change, the query engine code gathers information about
shards and tagsets by working with individual shards, collating the
information, and returning that to the client. It does not assume that any
particular shard is local, and accesses all shards through abstracted
Mappers, of which there are two types -- a Mapper type for Raw queries
and a second type for Aggregate queries. There are corresponding
Executors for each type of Mapper, but both types of Executors share the
same interface.
2015-07-15 12:54:55 -07:00
e4f2d8a6c4
Fixed httpd logger to get user from query params
2015-07-13 17:36:34 -04:00
72f52d44c9
making queries syntactically correct
2015-07-08 17:25:56 -06:00
6b8d3268e6
Fix code review comments
2015-07-07 11:41:12 -06:00
db63ada7db
Drop NaN and Inf values from graphite input
...
NaN is skipped by graphite. Inf is not a supported value for Influxdb.
2015-07-06 16:14:02 -06:00
b58df5344c
Use a single batcher for graphite service
...
Previously there was a batcher per connection and each batcher was
flushed when the connection was closed. This didn't have much of an
effect when multiple clients connected and disconnected since it would
flush the batch immediately. It also did not help UDP traffic.
Instead, there is now a shared batcher for the service so that multiple
connections will not cause frequent flushes.
2015-07-06 16:14:02 -06:00
4a71692b88
Allow extra tags using graphite default template
...
Fixes #3223
2015-07-06 16:14:02 -06:00
0b481d55f4
Fix graphite filter searching matching wrong template
...
Default template would get chosen in some cases when a matching filter
existed.
Fixes #3245
2015-07-06 16:14:02 -06:00
a3ab093996
Parse NaN as float
...
Fixes #3230
2015-07-06 16:14:01 -06:00
87c962d62d
Fix panics when collectd fails to start
...
If collectd fails to start, it can panic when close is called because
some variables have not be initialized yet.
2015-06-29 22:09:47 -06:00
5fccee3d16
Fix spelling errors in comments and strings.
2015-06-28 02:54:34 -04:00
ff61d88c10
Create UDP batcher before listening for UDP
...
Otherwise a packet may be received before the batcher has been
instantiated.
2015-06-26 10:17:33 -04:00
2c333c6c63
Fix markdown formatting
2015-06-25 22:08:13 -06:00
562d7cd37d
Handle timestamp special cases
...
If no timestamp is sent or the value -1 is sent, the current UTC
time is used.
2015-06-24 23:53:13 -06:00
c5a10cf93d
Use raw metric name when default template fails to match
2015-06-24 23:38:10 -06:00
fbfb90d66c
Code review fixes
2015-06-24 23:33:42 -06:00
320a951575
Fix default template being returned when partially matching
...
another filter.
2015-06-24 23:09:08 -06:00
ba7187f554
Add comments to graphite parser
2015-06-24 23:09:08 -06:00
a76e812b38
Add graphite parser benchmark
2015-06-24 23:09:08 -06:00
9ed71ad492
Add test for matching similar patterns
2015-06-24 23:09:08 -06:00
613b1d2eb7
Prevent duplicate filters in config
...
A filter should map directly to one template, allowing duplicate
filters is not supported.
2015-06-24 23:09:08 -06:00
f70eee6e7a
Add support for multiple measurement fields in templates
...
Provides a little more flexibility in controlling the parsed
metric names for metris like:
servers.localhost.cpu.cpu0.user
Previously, you could only use a single field like "cpu", "user"
or a wildcard to match "cpu.cpu0.user". You can now pull out "cpu"
and "user" and join them together in the metric name using a custom
separator character. By default this is ".".
2015-06-24 23:09:08 -06:00
a2a1956048
Use search tree for filter matching
...
This adds a sorted search tree for matchining filters to a template
more efficiently. Each filter is split on "." and each element is
added to the tree. Patterns with matching prefixes are added under
the same subtree.
2015-06-24 23:09:08 -06:00
98cbfdca51
Update tempalte format comment
2015-06-24 23:09:07 -06:00
b294930c95
Add graphite plugin readme
2015-06-24 23:09:07 -06:00
1ecf9b5d36
Fix validation failing when using a default template
2015-06-24 23:09:07 -06:00
dd0e6e5e02
Use strings.Fields to bef more forgiving of whitespace
2015-06-24 23:09:07 -06:00
fed8d67946
Add validation for graphite config templates and tags
2015-06-24 23:09:07 -06:00
b55981f090
Add support for per-template default tags
...
These are tags that can be added at the template level. They
will override any global tags and any parsed tags with the same
name from the metric will override these.
2015-06-24 23:09:07 -06:00
Philip O'Toole
Cameron Sparr
Paul Dix
Jason Wilder
Cory LaNou
Philip O'Toole
Antonio Murdaca
Rob Wilson
Jason Wilder
David Norton
Ben Johnson
Gunnar
Jon Seymour
gunnaraasen
Nathaniel Cook
Dimitrij Denissenko
Vorn Mom
Josh Horwitz
Nathaniel Cook
Todd Persen
Alexander Morozov
Sean Beckett
Joseph Crail