This commit changes raft so that term changes are made immediately and
term change signals are made afterward. Previously, election timeouts
were invalidated by incoming term changes which caused an election loop.
Stale term was also fixed and http/pprof was added too.
2015/04/08 22:27:01 no broker or server configured to handle messaging endpoints
2015/04/08 22:27:02 join: failed to connect data node: http://box296:9012: unable to join
2015/04/08 22:27:02 join: failed to connect data node to any specified server
There is a race when joining a data only node to a broker and another data only node between the
data node heartbeater and the join operation. If the heartbeater
fire before the join attempt, it's possible for the booting data node
to be selected as the first data node for redirection by the broker.
The join attempt would request a data node endpoint on the broker "/data_nodes"
but since the broker cannot handle it, it would redirect to a valid broker.
During this race, the broker would redirect the request back to the same server. If
this happens, the data node would get stuck and not be able to join because it's
still booting.
To work around this, the redirect is randonmized and the join calls will not attempt
to call itself and instead re-request the original URL. A better fix might be to
not start the heartbeater until after the datanode has joined or initialized.
This removes all join URLs from the config. To join a node to a
cluster, the URL of another member of the cluster should be passed
on the command line w/ the -join flag. The join URLs can now be
any node regardless of whether the node is a broker only or data
only node. At join time, the receiving node will redirect the
request to a valid broker or data node if it cannot handle the request
itself.
This is a pre-requisite for #1934. When running separate
broker and data nodes, you currently need to know what role
a host is performing. This complicates cluster setup in
that you must configure separate broker URLs and data node
URLs.
This change allows a broker only node to redirect data nodes endpoints
to a valid data node and a data only node to redirect broker
endpoints to a valid broker.
Philip O'Toole
Jason Wilder
Philip O'Toole
Ben Johnson