86 lines
3.1 KiB
Plaintext
86 lines
3.1 KiB
Plaintext
make integration_test verbose=on only=TestRestartServers
|
|
|
|
parts to create:
|
|
|
|
protobuf server
|
|
protobuf client
|
|
|
|
Coordinator has Datastore and ConcensusEngine
|
|
ConsensusEngine has a collection of ClusterServers (one is the localhost) and the ProtobufServer
|
|
|
|
ClusterServer has protobuf client (automatically tries to connect unless it's localhost)
|
|
ProtobufServer has a datastore to answer queries
|
|
|
|
|
|
ringLocations: [{servers: []}, {servers: []}]
|
|
|
|
startup:
|
|
create db
|
|
start protobuf server
|
|
start clusterConsensus
|
|
for each server, createClient and connect
|
|
|
|
write comes in:
|
|
split it into discreet ring locations (time series objects with points all on the same ring loc)
|
|
ring location is figured by hashing db, series, and time (rounded to 5 min interval)
|
|
proxy to one of the servers that have the ring location
|
|
|
|
On the server we proxied to...
|
|
log the write...
|
|
assign the operation a sequence number (should be an operation sequence number, have one per ring location)
|
|
assign points sequence numbers (can have one global incrementer)
|
|
log it -
|
|
<ring location><operation sequence><timestamp><db, series id> - serialized operation
|
|
|
|
attempt to write to other servers in ring
|
|
receiving server tracks for a given server id and ring location, what the last sequence was
|
|
for receiving server, if it's not the next in the sequence, set aside and request missing ops
|
|
|
|
write the value to key:
|
|
<db/series/column id><timestamp><sequence>
|
|
|
|
point sequence numbers:
|
|
first two bytes is the server id. Guaranteed unique in a cluster
|
|
the other six bytes are the sequence number.
|
|
|
|
every 5 minutes create a marker
|
|
<sequence marker prefix><ring location><time stamp> - <sequence number>
|
|
|
|
deleting series data
|
|
delete the values and do compactions.
|
|
for each ring location:
|
|
look up the sequence number for the time frame of the start of the deletion. Seek there and iterate over keys
|
|
if <db, series id> matches and timestamp in range, delete.
|
|
if timestamp out of range, break
|
|
run compaction for ring location range
|
|
|
|
every x seconds each server asks the other servers that own its ring locations to replay operations from a given sequence number
|
|
|
|
joining a server to a cluster:
|
|
figure out ring locations
|
|
ask servers to replay operations for ring locations from previous
|
|
tell old server to drop log and index data:
|
|
go through ring locations, read serialized operations, if it's a write then delete the corresponding values.
|
|
compact the ring log, let the points index compact gradually on its own
|
|
|
|
example of expanding cluster:
|
|
rf: 1, 2, 3
|
|
|
|
A 0-3333
|
|
B 3334 - 6666
|
|
C 6667 - 9999
|
|
|
|
A 0-2499
|
|
rf 1 [0-3333] -> [0-2499] del: (2499-3333)
|
|
rf 2 [6667-3333] -> [7500-2499] del: (2499-3333, 6667-7500)
|
|
rf 3 [3334-3333] -> [5000-2499] del: (2499-4999)
|
|
B 2500-4999
|
|
rf 1 [3334-6666] -> [2500-4999] del: (5000-6666)
|
|
rf 2 [0-6666] -> [0-4999] del: (5000-6666)
|
|
rf 3 [6667-6666] -> [7500-4999] del: (5000-7500)
|
|
C 5000-7499
|
|
rf 1 [6667-9999] -> [5000-7499] del: (7500-9999)
|
|
rf 2 [3334-9999] -> [2500-7499] del: (7500-9999)
|
|
rf 3 [0-9999] -> [0-7499] del: (7500-9999)
|
|
D 7500-9999
|
|
all butter... |