The interaction of continuous query service, the meta-store loading
and initializing raft state, and syncing node info could cause a
deadlock in some instances. There was an extra read-lock taken by isLeader()
when it already had a read-lock. Removing this extra lock fixes the startup
deadlock.
Fixes#3607
Instead of trying to start all the nodes with dynamic peer addresses
set, alwasy start one, then join the rest to this one. The SetPeers
in the test may be causing leadership changes and sporadic failures.
The raft.Shutdown() call was deadlocking if operations were still
being applied to the log sometimes. Change the shutdown behavior to
match how consul shuts down raft:
e37b5ecb69/consul/server.go (L471-L477)
* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
Renaming a host that is a raft peer member is pretty difficult but
we can special case single-node renames since we know all the member
in the cluster and we can update the peer store directly on all nodes
(just one).
Fixes#3632
If the -hostname flag is passed, the node will startup and be accessible from
remote nodes using the specified hostname. At startup, we attempt to update
the hostname if it's different. For data-only nodes, this is pretty straight-forward.
For nodes part of the raft cluster, it is much more complicated as the the cluster
must be up and stable (with a leader) for a the update to take place. The main
complication in this case is that the node starting up will have a different
hostname and will fail to take part of the raft cluster because each other node
does not have this new name in the it's raft peers list. Since this is very problematic
and very easy to break a cluster, this PR just aborts startup and alerts the operator that
some manual actions must be taken to update the raft peer on all raft members before
the hostname can be fully updated.
Fixes#3421
Hostnames were always being resolved to an IP address and the IP
address was used as the host address and raft peer address. There
was no way to use an actual hostname instead of an IP address.
Philip O'Toole
Jason Wilder
David Norton