2015/04/08 22:27:01 no broker or server configured to handle messaging endpoints
2015/04/08 22:27:02 join: failed to connect data node: http://box296:9012: unable to join
2015/04/08 22:27:02 join: failed to connect data node to any specified server
There is a race when joining a data only node to a broker and another data only node between the
data node heartbeater and the join operation. If the heartbeater
fire before the join attempt, it's possible for the booting data node
to be selected as the first data node for redirection by the broker.
The join attempt would request a data node endpoint on the broker "/data_nodes"
but since the broker cannot handle it, it would redirect to a valid broker.
During this race, the broker would redirect the request back to the same server. If
this happens, the data node would get stuck and not be able to join because it's
still booting.
To work around this, the redirect is randonmized and the join calls will not attempt
to call itself and instead re-request the original URL. A better fix might be to
not start the heartbeater until after the datanode has joined or initialized.
3 was fairly arbitrary and would cause errors such as:
2015/04/08 14:01:12 join: failed to connect data node: {http <nil> influxdb.local:8191 }: unable to join
2015/04/08 14:01:12 join: failed to connect data node to any specified server
in the tests. This can happen when the nodes are slow to startup. The limit is set
arbitarily higher to avoid this error but still give up if it can't connect
after a minute.
If a node is restarted and it had already joined the cluster,
ignore and log that the join urls are being ignored and existing
cluster state will be used.
When starting multiple servers concurrently, they can race to connect
to each other. This change just has the join attempts retry to make
cluster setup easier.
This removes all join URLs from the config. To join a node to a
cluster, the URL of another member of the cluster should be passed
on the command line w/ the -join flag. The join URLs can now be
any node regardless of whether the node is a broker only or data
only node. At join time, the receiving node will redirect the
request to a valid broker or data node if it cannot handle the request
itself.
To add a new data node, it currently needs a broker
and another data node to join. Temporarily adding
a JoinURLs option to the Data node section so a
standalone data node can be created but the intent is
that this will be removed.
Ideally, the the joinURL could point to either a data node
or a broker and it would get the required URLs from that
host but that is not possible currently.
This can happen, though is very unlikely. If this node receives encoded
data, to be written to disk, and is queried for that data before its
metastore is updated, there will be no field mapping for the data during
decode. All this can happen because data is encoded by the node that first
received the write request, not the node that actually writes the data to
disk. So if this happens, skip the data.
This is a pre-requisite for #1934. When running separate
broker and data nodes, you currently need to know what role
a host is performing. This complicates cluster setup in
that you must configure separate broker URLs and data node
URLs.
This change allows a broker only node to redirect data nodes endpoints
to a valid data node and a data only node to redirect broker
endpoints to a valid broker.
Refactored query engine to have different processing pipeline for raw queries. This enables queries that have a large offset to not keep everything in memory. It also makes it so that queries against raw data that have a limit will only p
rocess up to that limit and then bail out.
Raw data queries will only read up to a certain point in the map phase before yielding to the engine for further processing.
Fixes#2029 and fixes#2030
This sends data node urls via the broker heartbeat from each data
node. The urls are tracked on the broker to support simpler
cluster setup as well as distributed queries.
This fixes all "golint" warnings in the top-level "influxdb" package:
database.go:316:2: can probably use "var tagStrings []string" instead
database.go:1077:6: exported type RetentionPolicies should have comment or be unexported
server.go:337:1: exported method Server.StartSelfMonitoring should have comment or be unexported
server.go:345:3: can probably use "var points []Point" instead
server.go:433:2: can probably use "var groups []group" instead
server.go:498:2: can probably use "var groups []group" instead
server.go:1417:1: comment on exported method Server.CreateRetentionPolicyIfNotExists should be of the form "CreateRetentionPolicyIfNotExists ..."
server.go:1899:1: exported method Server.DropMeasurement should have comment or be unexported
server.go:2153:10: if block ends with a return statement, so drop this else and outdent its block
server.go:2695:2: can probably use "var rows []*influxql.Row" instead
server.go:2876:11: if block ends with a return statement, so drop this else and outdent its block
server.go:3804:1: exported method Server.StartReportingLoop should have comment or be unexported
tx.go:133:4: can probably use "var mappers []influxql.Mapper" instead
tx.go:231:1: exported method LocalMapper.Open should have comment or be unexported
tx.go:254:1: exported method LocalMapper.Close should have comment or be unexported
tx.go:348:1: exported method LocalMapper.Next should have comment or be unexported
This allows historical monitoring of all diagnostics. Of particular
interest will be GC performance, memory usage, number of shards and
shard groups on the cluster, index points, amongst other important data.
This will make it easier to use these common diagnostics function in
other components, like the Broker, as well as minimizing growth of the
server type, in terms of SLOC.
This new diags code can be converted to InfluxQL rows, for easy display
and re-writes back to the database for self-monitoring.
This commit adds the "influxd restore" command to the CLI. This allows
a snapshot that has been produced by "influxd backup" to be restored
to a config location and the broker and raft directories will be
bootstrapped based on the state of the snapshot.
David Norton
Jason Wilder
Cory LaNou
Philip O'Toole
Paul Dix
Philip O'Toole
Todd Persen
Jari Sukanen
Dejan Golja
Julius Volz
Todd Persen
Ben Johnson
Simon Kern
Giulio Iotti