Commit Graph

158 Commits (b6508348482eab126825855423c7aecb48a31813)

Author SHA1 Message Date
Philip O'Toole b1c5a36b0d Merge pull request #4846 from jsternberg/feature/allow-nan-in-graphite-service
Allow NaN as a valid value on the graphite service
2015-11-20 08:09:51 -08:00
Jonathan A. Sternberg 5fa36639db Allow NaN as a valid value on the graphite service
The canonical graphite implementation will read and discard NaN values
instead of throwing an error when reading on the line receiver protocol.
Since this is the default behavior for graphite, InfluxDB should have
the same behavior for compatibility.

Previously, a NaN value would result in an error printed to the console.
When you have a large number of NaN values being sent every minute, this
results in the log file filling with useless messages.
2015-11-20 11:07:25 -05:00
Philip O'Toole d86a02eeab Fix minor typo in variable name 2015-11-20 07:48:42 -08:00
Philip O'Toole bc1091ec31 Revert "Merge pull request #4684 from nkatsaros/default-config"
This reverts commit 87441584f3, reversing
changes made to a14829bc1d.
2015-11-11 09:29:06 -08:00
Cameron Sparr 9625953d3e Add UDP OS buffer size recommendations 2015-11-10 11:00:11 -07:00
Cameron Sparr 727b9f6d8b Documentation updates for UDP, graphite, collectd listeners 2015-11-10 10:35:21 -07:00
Cameron Sparr 0a683144fb Update collectd and graphite UDP listeners with perf enhancements
closes #4678
2015-11-10 10:35:21 -07:00
Nicholas Katsaros 4f8b0aca9a Add graphite and udp services to the default config generator 2015-11-05 16:42:21 -05:00
Philip O'Toole de7919240f Migrate internal stats to consistent names
Go style -- and existing runtime stats -- do not use underscores, but
instead use camel case. This change makes the internal stats adhere to
that convention.
2015-10-28 21:07:45 -07:00
Jason Wilder 0926b19e6b Prevent creating points with NaN float values
Float values are not supported in the existing engine and the tsm1
engines.  This changes NewPoint to return an error if a field value
contains a NaN field.  It also allows us to validate fields to prevent
other unsupported types from sneaking in through other input plugins.
2015-10-27 17:12:52 -06:00
bwhaley 8aa79c551f Update README.md
Clarify use of filters with multiple consecutive wildcards
2015-10-24 15:01:44 -07:00
Philip O'Toole ff18bf7213 Make Open() and Close() on Graphite sync'ed
This will ensure that these operations don't run concurrently. This
change also ensures nil batchers are not closed.

Fixes issue #4494.
2015-10-19 11:13:31 -07:00
Sean Beckett 5ab86f7578 Update README.md 2015-10-13 16:56:37 -07:00
Sean Beckett ed7b9f7485 tag name --> tag key 2015-10-13 16:41:57 -07:00
Cameron Sparr 2add55107e Fix graphite parser merge error, nargs 2015-10-08 11:15:03 -06:00
Cameron Sparr 3bea25b428 graphite parser: apply tags from the Parser on the template 2015-10-08 10:56:13 -06:00
Rob Wilson f3e3bf7a0e typo 2015-10-07 21:25:49 +01:00
Rob Wilson 5815e0b0ee updated documentation 2015-10-07 21:24:05 +01:00
Rob Wilson d8ac746703 correct formatting 2015-10-07 20:35:05 +01:00
Rob Wilson 5fd8777c56 add tests 2015-10-07 20:32:10 +01:00
Rob Wilson a27186fb7a raise exception when field keyword is specified multiple times 2015-10-07 20:31:46 +01:00
Rob Wilson bcd6c06173 Merge remote-tracking branch 'upstream/master' into graphite-template-custom-field
Conflicts:
	services/graphite/parser.go
2015-10-07 17:48:34 +01:00
Cameron Sparr 883d32cfd0 Add public function to graphite parser to apply template 2015-10-06 17:42:36 -06:00
Philip O'Toole 9de3125f6b Graphite TCP should not block system shutdown
With this change Graphite TCP connections are tracked on a per-service
basis. This allows a closing Graphite service to first shutdown any
active connections, thereby unblocking the rest of shutdowm.

This work exposed small shortcomings with the existing Diagnostics
system and that code has alse been tweaked.

Fixes issue #4017
2015-09-24 14:08:38 -07:00
Rob Wilson ef35d6dcc2 formatting 2015-09-21 12:26:43 +01:00
Rob Wilson 27c1cc23fd Working prototype.. 2015-09-21 12:18:19 +01:00
Rob Wilson 9121b422f8 comment out tests for now.. 2015-09-21 10:47:42 +01:00
Rob Wilson 20e4fdfa9a allow specifying fieldname in graphite template 2015-09-20 21:17:50 +01:00
Cory LaNou d19a510ad2 refactor Points and Rows to dedicated packages 2015-09-16 15:33:08 -05:00
Philip O'Toole 02fcaf853d Add node re Graphite configuration
[ci skip]
2015-09-08 23:22:34 -07:00
Philip O'Toole 5373f263a3 Add pending control to batcher
With this change, the generic batcher used by many inputs can now be
buffered. Testing shows that this performance of the Graphite input by
10-100%, with the biggest improvements at lower numbers of connections.
2015-09-08 19:32:00 -07:00
Jason Wilder 73510a0a68 Fix invalid time stamp in graphite metric causes panic
If a timestamp was larger than the max epoch value was sent via
graphite it would cause the timestamp to overflow when it was
marshaled/unmarshaled back from the raft log.  The overflow cause
the shard group to get created with the wrong timestamp which cause
a panic when writing the point.  The panic was caused because the
timestamp that were supposed to exists in a map created by MapShards
did not actually exist so a nil ShardGroup was used.

The change prevents creating the point with an invalid timestamp.  Since
graphite using a timestamp in seconds, the maximum range is known and
can be prevented.  This also adds a check for the minimum range as well.

Fixes #3785
2015-09-08 10:07:47 -06:00
Philip O'Toole 332ce6481d Removed unused Graphite NewConfig
This function is not helpful for sections of the config that support
multiple instances.
2015-09-08 08:32:19 -07:00
Philip O'Toole bbc103305b Support multiple Graphite inputs
Fixes issue #3636
2015-09-06 21:33:46 -07:00
Philip O'Toole fa29e12222 Shutdown UDP Graphite on SIGTERM
Service.Close() had no way of closing the UDP Conn. This change makes
the UDP an attribute of the server, so Close() can access it.
2015-09-05 00:30:59 -07:00
Philip O'Toole 89bc392ec4 Access expvar directly from monitor
expvar map is already global so access it directly. This simplifies the
code and makes it much eaisier to use from other modules.
2015-09-04 09:45:24 -07:00
Philip O'Toole 6ad35e23e9 Integrate code review feedback 2015-09-03 20:50:54 -07:00
Philip O'Toole d58532d844 Add Graphite diagnostics
Graphite diagnostics currently show TCP connections.
2015-09-03 20:50:54 -07:00
Philip O'Toole e07432c59f Implement diagnostics support
This change adds support for diagnostics by decomposing the existing
interface into two interfaces -- one for stats, and the other for
diags. It also adds some basic monitor of system, network, and the Go
runtime.
2015-09-03 20:50:54 -07:00
Philip O'Toole 4e2ee1ea70 Rename MonitorService to just Monitor
monitor is not a service, it has more in common with meta, since it
provides functionality to the query layer. This names makes this
clearer.
2015-09-02 15:07:30 -07:00
Philip O'Toole 9df17409d3 Use monitor service with Graphite 2015-09-01 15:21:36 -07:00
Philip O'Toole 6193226ce8 Revert "Merge pull request #3771 from influxdb/tcp_graphite_timeout"
This reverts commit d7f646f7a4, reversing
changes made to d6f9903f10.

Conflicts:
	CHANGELOG.md

Fixes issue #3809
2015-08-24 10:53:14 -07:00
Philip O'Toole d7f646f7a4 Merge pull request #3771 from influxdb/tcp_graphite_timeout
Close idle Graphite TCP connections
2015-08-20 17:08:17 -07:00
Philip O'Toole 50b0f67290 Add Graphite TCP timeout tests 2015-08-20 15:46:08 -07:00
Philip 4930a6d8bb Start adding timeouts to TCP Graphite input 2015-08-20 15:10:22 -07:00
Jason Wilder afe1f598ca Cache name and fields if requested
Through profiling of writes, point.Fields() and point.Name() were called
repeatedly in PointsWriter and the Shard.  These calls are somewhat expensive
when writing large batches so we can cache them to avoid wasting CPU cycles.

Using influx_stress with default settings

Before:
  Wrote 10000000 points at average rate of 202570
  Average response time:  235.450355ms

After:
  Wrote 10000000 points at average rate of 246120
  Average response time:  182.881008ms
2015-08-20 15:48:38 -06:00
Philip 8e51064db1 Log Graphite batch size and timeout 2015-08-20 11:23:09 -07:00
Gunnar cf5ac2603d Fix Graphite README typo
Fixes #3727
2015-08-19 07:53:29 -07:00
Jason Wilder 668181d275 Make log statements more consistent
* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
2015-08-13 10:01:42 -06:00
Philip O'Toole 089d947bf3 Shutdown Graphite listener first during Close()
Without this the WaitGroup was not fully decremented as the Accept()
call on the listener never exited, and Wait() then never exited.
2015-08-12 12:49:58 -07:00
Philip O'Toole 966dee7559 Set sensible Graphite batching defaults 2015-08-11 18:34:36 -07:00
Vorn Mom 38387ba6b7 Fixed typo. 2015-07-24 19:41:46 -04:00
Vorn Mom 5b50002728 Fixed typo in README 2015-07-24 17:07:41 -04:00
Alexander Morozov 675eacbf2c Fix style issues with else
In go it's better to just continue flow without "else", if it is return in
"if" statement.

Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-07-17 11:10:23 -07:00
Jason Wilder 6b8d3268e6 Fix code review comments 2015-07-07 11:41:12 -06:00
Jason Wilder db63ada7db Drop NaN and Inf values from graphite input
NaN is skipped by graphite.  Inf is not a supported value for Influxdb.
2015-07-06 16:14:02 -06:00
Jason Wilder b58df5344c Use a single batcher for graphite service
Previously there was a batcher per connection and each batcher was
flushed when the connection was closed.  This didn't have much of an
effect when multiple clients connected and disconnected since it would
flush the batch immediately.  It also did not help UDP traffic.

Instead, there is now a shared batcher for the service so that multiple
connections will not cause frequent flushes.
2015-07-06 16:14:02 -06:00
Jason Wilder 4a71692b88 Allow extra tags using graphite default template
Fixes #3223
2015-07-06 16:14:02 -06:00
Jason Wilder 0b481d55f4 Fix graphite filter searching matching wrong template
Default template would get chosen in some cases when a matching filter
existed.

Fixes #3245
2015-07-06 16:14:02 -06:00
Jason Wilder a3ab093996 Parse NaN as float
Fixes #3230
2015-07-06 16:14:01 -06:00
Jason Wilder 2c333c6c63 Fix markdown formatting 2015-06-25 22:08:13 -06:00
Jason Wilder 562d7cd37d Handle timestamp special cases
If no timestamp is sent or the value -1 is sent, the current UTC
time is used.
2015-06-24 23:53:13 -06:00
Jason Wilder c5a10cf93d Use raw metric name when default template fails to match 2015-06-24 23:38:10 -06:00
Jason Wilder fbfb90d66c Code review fixes 2015-06-24 23:33:42 -06:00
Jason Wilder 320a951575 Fix default template being returned when partially matching
another filter.
2015-06-24 23:09:08 -06:00
Jason Wilder ba7187f554 Add comments to graphite parser 2015-06-24 23:09:08 -06:00
Jason Wilder a76e812b38 Add graphite parser benchmark 2015-06-24 23:09:08 -06:00
Jason Wilder 9ed71ad492 Add test for matching similar patterns 2015-06-24 23:09:08 -06:00
Jason Wilder 613b1d2eb7 Prevent duplicate filters in config
A filter should map directly to one template, allowing duplicate
filters is not supported.
2015-06-24 23:09:08 -06:00
Jason Wilder f70eee6e7a Add support for multiple measurement fields in templates
Provides a little more flexibility in controlling the parsed
metric names for metris like:

  servers.localhost.cpu.cpu0.user

Previously, you could only use a single field like "cpu", "user"
or a wildcard to match "cpu.cpu0.user".  You can now pull out "cpu"
and "user" and join them together in the metric name using a custom
separator character.  By default this is ".".
2015-06-24 23:09:08 -06:00
Jason Wilder a2a1956048 Use search tree for filter matching
This adds a sorted search tree for matchining filters to a template
more efficiently.  Each filter is split on "." and each element is
added to the tree.  Patterns with matching prefixes are added under
the same subtree.
2015-06-24 23:09:08 -06:00
Jason Wilder 98cbfdca51 Update tempalte format comment 2015-06-24 23:09:07 -06:00
Jason Wilder b294930c95 Add graphite plugin readme 2015-06-24 23:09:07 -06:00
Jason Wilder 1ecf9b5d36 Fix validation failing when using a default template 2015-06-24 23:09:07 -06:00
Jason Wilder dd0e6e5e02 Use strings.Fields to bef more forgiving of whitespace 2015-06-24 23:09:07 -06:00
Jason Wilder fed8d67946 Add validation for graphite config templates and tags 2015-06-24 23:09:07 -06:00
Jason Wilder b55981f090 Add support for per-template default tags
These are tags that can be added at the template level.  They
will override any global tags and any parsed tags with the same
name from the metric will override these.
2015-06-24 23:09:07 -06:00
Jason Wilder 9cd82ae316 Add support for global tags
These are tags that can be add to all metrics.
2015-06-24 23:09:07 -06:00
Jason Wilder cab9e36111 Add basic template filtering support
This filter implementation is fairly naive and won't scale well
to large numbers of templates and filters.  It will be replaced
with a trie-based approach in the future.
2015-06-24 23:09:07 -06:00
Jason Wilder e5fbf249ff Remove DecodeNameAndTags func 2015-06-24 23:09:07 -06:00
Jason Wilder 46046c6d32 Add matcher for match a metric to a template
If no template matches, use a default template that just matches
the whole metric name as the measurement.
2015-06-24 23:09:07 -06:00
Jason Wilder 9bcbbd9875 Rename matcher to template 2015-06-24 23:09:07 -06:00
Jason Wilder d69a21c4e4 Allow skipping fields in templates 2015-06-24 23:09:07 -06:00
Jason Wilder a7d4d97743 Convert template matching to matcher struct 2015-06-24 23:09:06 -06:00
Jason Wilder d132263f08 Add support for measurement* 2015-06-24 23:09:06 -06:00
Jason Wilder d539b23817 Move graphite parser to separate file 2015-06-24 23:09:06 -06:00
Can ZHANG 2a383e6858 Fix unit tests for graphite 2015-06-24 23:09:06 -06:00
Can ZHANG c130efb5e2 Add fields to config metric name schema of graphite 2015-06-24 23:09:06 -06:00
Philip O'Toole 4dff5f48aa Export WaitForLeader on MetaStore
This exported function can then be used by Services and
server-reporting, so those components don't make progress until the
cluster is ready.
2015-06-11 11:10:07 -07:00
Philip O'Toole e84f661f7e Use defaults for Graphite input where necessary 2015-06-10 20:38:17 -07:00
Philip O'Toole dddaf46b77 Update CHANGELOG 2015-06-10 20:02:37 -07:00
Philip O'Toole 57ce67b97b Ensure target Graphite database exists 2015-06-10 18:03:06 -07:00
Philip O'Toole d942250c5e Merge pull request #2862 from runner-mei/connection_closed2
Make InfluxDB win32 friendly, fix unit test is failed on the windows
2015-06-10 15:58:57 -07:00
Cory LaNou 8a5cf394d8 add ability to silence logging for testing 2015-06-10 10:27:57 -05:00
runner.mei 015adc949c Make InfluxDB win32 friendly, fix unit test is failed on the windows 2015-06-10 07:29:08 +08:00
Philip O'Toole 8eea2b3092 Allow Server creation to return errors 2015-06-09 14:21:12 -07:00
Philip O'Toole 952fb49368 Move parsing consistency levels to cluster package
Errors parsing the consistency level are indicated by returning a nil
Graphite input.
2015-06-09 14:21:12 -07:00
Philip O'Toole 4769d65f23 Final Graphite test code fixes 2015-06-08 20:39:56 -07:00
Philip O'Toole 9d628320c2 Rename test file to service_test.go 2015-06-08 20:39:42 -07:00
Philip O'Toole 4d01c8ad41 Add test of Graphite UDP input 2015-06-08 20:39:42 -07:00