WaitForDataChanged is used to block a goroutine so it can be woken
up when the metastore has been updated. This makes it so clients
to the metastore do not need to poll the metastore repeatedly to
watch for changes. This was only working for local raft nodes, but
was supposed to work for remote nodes as well.
The interaction of continuous query service, the meta-store loading
and initializing raft state, and syncing node info could cause a
deadlock in some instances. There was an extra read-lock taken by isLeader()
when it already had a read-lock. Removing this extra lock fixes the startup
deadlock.
Fixes#3607
The raft.Shutdown() call was deadlocking if operations were still
being applied to the log sometimes. Change the shutdown behavior to
match how consul shuts down raft:
e37b5ecb69/consul/server.go (L471-L477)
Renaming a host that is a raft peer member is pretty difficult but
we can special case single-node renames since we know all the member
in the cluster and we can update the peer store directly on all nodes
(just one).
Fixes#3632
If the -hostname flag is passed, the node will startup and be accessible from
remote nodes using the specified hostname. At startup, we attempt to update
the hostname if it's different. For data-only nodes, this is pretty straight-forward.
For nodes part of the raft cluster, it is much more complicated as the the cluster
must be up and stable (with a leader) for a the update to take place. The main
complication in this case is that the node starting up will have a different
hostname and will fail to take part of the raft cluster because each other node
does not have this new name in the it's raft peers list. Since this is very problematic
and very easy to break a cluster, this PR just aborts startup and alerts the operator that
some manual actions must be taken to update the raft peer on all raft members before
the hostname can be fully updated.
Fixes#3421
Hostnames were always being resolved to an IP address and the IP
address was used as the host address and raft peer address. There
was no way to use an actual hostname instead of an IP address.
Jason Wilder
Philip O'Toole
Cory LaNou
Jason Wilder