Instead of trying to start all the nodes with dynamic peer addresses
set, alwasy start one, then join the rest to this one. The SetPeers
in the test may be causing leadership changes and sporadic failures.
The raft.Shutdown() call was deadlocking if operations were still
being applied to the log sometimes. Change the shutdown behavior to
match how consul shuts down raft:
e37b5ecb69/consul/server.go (L471-L477)
* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
Renaming a host that is a raft peer member is pretty difficult but
we can special case single-node renames since we know all the member
in the cluster and we can update the peer store directly on all nodes
(just one).
Fixes#3632
If the -hostname flag is passed, the node will startup and be accessible from
remote nodes using the specified hostname. At startup, we attempt to update
the hostname if it's different. For data-only nodes, this is pretty straight-forward.
For nodes part of the raft cluster, it is much more complicated as the the cluster
must be up and stable (with a leader) for a the update to take place. The main
complication in this case is that the node starting up will have a different
hostname and will fail to take part of the raft cluster because each other node
does not have this new name in the it's raft peers list. Since this is very problematic
and very easy to break a cluster, this PR just aborts startup and alerts the operator that
some manual actions must be taken to update the raft peer on all raft members before
the hostname can be fully updated.
Fixes#3421
Hostnames were always being resolved to an IP address and the IP
address was used as the host address and raft peer address. There
was no way to use an actual hostname instead of an IP address.
Non-raft nodes need to be notifified when the metastore changes. For
example, a database could be dropped on node 1 (non-raft) and node 2
would not know. Since queries for that database would not be a cache
miss, node 2 would not get updated.
To propogate changes to non-raft nodes, each non-raft node maintains
a blocking connection to a raft node that blocks until a metadata
change occurs. When the change is triggered, the updated metadata
is returned to the client and the client idempotently updates its local
cache. It then reconnects and waits for another change. This is
similar watches in zookeeper or etcd. Since the blocking request is
always recreated, it also serves as a polling mechanism that will retry
another raft member if the current connection is lost.
Some errors would not be returned to the client because something
failed before we could create the appropriate respone. For these
cases, a general error response is returned.
Philip O'Toole
Jason Wilder