* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
Renaming a host that is a raft peer member is pretty difficult but
we can special case single-node renames since we know all the member
in the cluster and we can update the peer store directly on all nodes
(just one).
Fixes#3632
If the -hostname flag is passed, the node will startup and be accessible from
remote nodes using the specified hostname. At startup, we attempt to update
the hostname if it's different. For data-only nodes, this is pretty straight-forward.
For nodes part of the raft cluster, it is much more complicated as the the cluster
must be up and stable (with a leader) for a the update to take place. The main
complication in this case is that the node starting up will have a different
hostname and will fail to take part of the raft cluster because each other node
does not have this new name in the it's raft peers list. Since this is very problematic
and very easy to break a cluster, this PR just aborts startup and alerts the operator that
some manual actions must be taken to update the raft peer on all raft members before
the hostname can be fully updated.
Fixes#3421
Hostnames were always being resolved to an IP address and the IP
address was used as the host address and raft peer address. There
was no way to use an actual hostname instead of an IP address.
Non-raft nodes need to be notifified when the metastore changes. For
example, a database could be dropped on node 1 (non-raft) and node 2
would not know. Since queries for that database would not be a cache
miss, node 2 would not get updated.
To propogate changes to non-raft nodes, each non-raft node maintains
a blocking connection to a raft node that blocks until a metadata
change occurs. When the change is triggered, the updated metadata
is returned to the client and the client idempotently updates its local
cache. It then reconnects and waits for another change. This is
similar watches in zookeeper or etcd. Since the blocking request is
always recreated, it also serves as a polling mechanism that will retry
another raft member if the current connection is lost.
Some errors would not be returned to the client because something
failed before we could create the appropriate respone. For these
cases, a general error response is returned.
Nodes that are not part of the raft cluster will not reliably know who the
current raft cluster leader is. To make communication simpler, proxy all
rpc and raft calls to the current raft leader if a non-leader receives one.
This adds some basic ability to join a node to an existing cluster. It
uses a rpc layer to initiate a join request to an existing memeber. The
response indicates whether the joining node should take part in the raft
cluster and who it's peers should be. If raft should not be started, the
peers are the addresses of the current raft members that it should delegate
consensus operations.
To keep the meta store implementation agnostic of whether it's running
a local raft or not, a consensusStrategy type was also added.
Jason Wilder