This is to prevents users from putting their system into an awkward
state. It is a policy that all databases must have at least a default
retention policy.
Fixes issue #3699.
The meta-store would open but may not have finished loading the raft log. If write
requests came in, they could fail or be dropped because of missing shard group
info. This change makes the meta store only return after it has found the leader
and is really ready.
This change also fixed a race in the ClusterRestart test that may be causing it
to fail sporadically.
Fixes#3677#3960
This commit adds the ability to list all shards in the cluster
and return their id, start time, end time, expiry time, and
owner ids. Shards are grouped by database.
Fixes#3562
This commit converts meta.ShardInfo.OwnerIDs from a slice of ids
to a slice of objects. This is to support adding statuses for a
shard for a given node. For example, a node may have a shard
assigned to it but it is currently copying the shard and is not
ready to serve data for it.
The old `OwnerIDs` is marked as deprecated, however, the code
still supports loading from older protobuf-encoded data.
The interaction of continuous query service, the meta-store loading
and initializing raft state, and syncing node info could cause a
deadlock in some instances. There was an extra read-lock taken by isLeader()
when it already had a read-lock. Removing this extra lock fixes the startup
deadlock.
Fixes#3607
Instead of trying to start all the nodes with dynamic peer addresses
set, alwasy start one, then join the rest to this one. The SetPeers
in the test may be causing leadership changes and sporadic failures.
The raft.Shutdown() call was deadlocking if operations were still
being applied to the log sometimes. Change the shutdown behavior to
match how consul shuts down raft:
e37b5ecb69/consul/server.go (L471-L477)
* Capitalize first letter of message
* Log all services staring consistently
* Remove some extraneous log statements in meta.Store
* Log data dirs for meta, data and hinted handoff
Renaming a host that is a raft peer member is pretty difficult but
we can special case single-node renames since we know all the member
in the cluster and we can update the peer store directly on all nodes
(just one).
Fixes#3632
If the -hostname flag is passed, the node will startup and be accessible from
remote nodes using the specified hostname. At startup, we attempt to update
the hostname if it's different. For data-only nodes, this is pretty straight-forward.
For nodes part of the raft cluster, it is much more complicated as the the cluster
must be up and stable (with a leader) for a the update to take place. The main
complication in this case is that the node starting up will have a different
hostname and will fail to take part of the raft cluster because each other node
does not have this new name in the it's raft peers list. Since this is very problematic
and very easy to break a cluster, this PR just aborts startup and alerts the operator that
some manual actions must be taken to update the raft peer on all raft members before
the hostname can be fully updated.
Fixes#3421
Hostnames were always being resolved to an IP address and the IP
address was used as the host address and raft peer address. There
was no way to use an actual hostname instead of an IP address.
Non-raft nodes need to be notifified when the metastore changes. For
example, a database could be dropped on node 1 (non-raft) and node 2
would not know. Since queries for that database would not be a cache
miss, node 2 would not get updated.
To propogate changes to non-raft nodes, each non-raft node maintains
a blocking connection to a raft node that blocks until a metadata
change occurs. When the change is triggered, the updated metadata
is returned to the client and the client idempotently updates its local
cache. It then reconnects and waits for another change. This is
similar watches in zookeeper or etcd. Since the blocking request is
always recreated, it also serves as a polling mechanism that will retry
another raft member if the current connection is lost.
linearb
Philip O'Toole
Cory LaNou
Philip O'Toole
Jason Wilder
Ben Johnson
David Norton