Merge branch 'master' of github.com:influxdb/influxdb into export
Joseph Rothrock
commit
5156a10e9e
|
|
@ -7,6 +7,7 @@
|
|||
- [#1971](https://github.com/influxdb/influxdb/pull/1971): Fix leader id initialization.
|
||||
- [#1975](https://github.com/influxdb/influxdb/pull/1975): Require `q` parameter for query endpoint.
|
||||
- [#1969](https://github.com/influxdb/influxdb/pull/1969): Print loaded config.
|
||||
- [#1987](https://github.com/influxdb/influxdb/pull/1987): Fix config print startup statement for when no config is provided.
|
||||
|
||||
## v0.9.0-rc12 [2015-03-15]
|
||||
|
||||
|
|
|
|||
|
|
@ -104,7 +104,7 @@ func execRun(args []string) {
|
|||
log.SetFlags(log.LstdFlags)
|
||||
writePIDFile(*pidPath)
|
||||
|
||||
if configPath != nil {
|
||||
if *configPath == "" {
|
||||
log.Println("No config provided, using default settings")
|
||||
}
|
||||
config := parseConfig(*configPath, *hostname)
|
||||
|
|
|
|||
|
|
@ -1,6 +0,0 @@
|
|||
# Questions on Github issues
|
||||
|
||||
Hi, github issues are reserved for bugs and/or features
|
||||
discussions. Please email your questions to the mailing list or hit us
|
||||
up on the irc channel. You can find more info on the
|
||||
[community page](http://influxdb.com/community/)
|
||||
|
|
@ -1,86 +0,0 @@
|
|||
make integration_test verbose=on only=TestRestartServers
|
||||
|
||||
parts to create:
|
||||
|
||||
protobuf server
|
||||
protobuf client
|
||||
|
||||
Coordinator has Datastore and ConcensusEngine
|
||||
ConsensusEngine has a collection of ClusterServers (one is the localhost) and the ProtobufServer
|
||||
|
||||
ClusterServer has protobuf client (automatically tries to connect unless it's localhost)
|
||||
ProtobufServer has a datastore to answer queries
|
||||
|
||||
|
||||
ringLocations: [{servers: []}, {servers: []}]
|
||||
|
||||
startup:
|
||||
create db
|
||||
start protobuf server
|
||||
start clusterConsensus
|
||||
for each server, createClient and connect
|
||||
|
||||
write comes in:
|
||||
split it into discreet ring locations (time series objects with points all on the same ring loc)
|
||||
ring location is figured by hashing db, series, and time (rounded to 5 min interval)
|
||||
proxy to one of the servers that have the ring location
|
||||
|
||||
On the server we proxied to...
|
||||
log the write...
|
||||
assign the operation a sequence number (should be an operation sequence number, have one per ring location)
|
||||
assign points sequence numbers (can have one global incrementer)
|
||||
log it -
|
||||
<ring location><operation sequence><timestamp><db, series id> - serialized operation
|
||||
|
||||
attempt to write to other servers in ring
|
||||
receiving server tracks for a given server id and ring location, what the last sequence was
|
||||
for receiving server, if it's not the next in the sequence, set aside and request missing ops
|
||||
|
||||
write the value to key:
|
||||
<db/series/column id><timestamp><sequence>
|
||||
|
||||
point sequence numbers:
|
||||
first two bytes is the server id. Guaranteed unique in a cluster
|
||||
the other six bytes are the sequence number.
|
||||
|
||||
every 5 minutes create a marker
|
||||
<sequence marker prefix><ring location><time stamp> - <sequence number>
|
||||
|
||||
deleting series data
|
||||
delete the values and do compactions.
|
||||
for each ring location:
|
||||
look up the sequence number for the time frame of the start of the deletion. Seek there and iterate over keys
|
||||
if <db, series id> matches and timestamp in range, delete.
|
||||
if timestamp out of range, break
|
||||
run compaction for ring location range
|
||||
|
||||
every x seconds each server asks the other servers that own its ring locations to replay operations from a given sequence number
|
||||
|
||||
joining a server to a cluster:
|
||||
figure out ring locations
|
||||
ask servers to replay operations for ring locations from previous
|
||||
tell old server to drop log and index data:
|
||||
go through ring locations, read serialized operations, if it's a write then delete the corresponding values.
|
||||
compact the ring log, let the points index compact gradually on its own
|
||||
|
||||
example of expanding cluster:
|
||||
rf: 1, 2, 3
|
||||
|
||||
A 0-3333
|
||||
B 3334 - 6666
|
||||
C 6667 - 9999
|
||||
|
||||
A 0-2499
|
||||
rf 1 [0-3333] -> [0-2499] del: (2499-3333)
|
||||
rf 2 [6667-3333] -> [7500-2499] del: (2499-3333, 6667-7500)
|
||||
rf 3 [3334-3333] -> [5000-2499] del: (2499-4999)
|
||||
B 2500-4999
|
||||
rf 1 [3334-6666] -> [2500-4999] del: (5000-6666)
|
||||
rf 2 [0-6666] -> [0-4999] del: (5000-6666)
|
||||
rf 3 [6667-6666] -> [7500-4999] del: (5000-7500)
|
||||
C 5000-7499
|
||||
rf 1 [6667-9999] -> [5000-7499] del: (7500-9999)
|
||||
rf 2 [3334-9999] -> [2500-7499] del: (7500-9999)
|
||||
rf 3 [0-9999] -> [0-7499] del: (7500-9999)
|
||||
D 7500-9999
|
||||
all butter...
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
Just some notes about requirements, design, and clustering.
|
||||
|
||||
Scalable datastore for metrics, events, and real-time analytics
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
* horizontally scalable
|
||||
* http interface
|
||||
* udp interface (low priority)
|
||||
* persistent
|
||||
* metadata for time series (low priority)
|
||||
* perform functions quickly (count, unique, sum, etc.)
|
||||
* group by time intervals (e.g. count ticks every 5 minutes)
|
||||
* joining multiple time series to generate new timeseries
|
||||
* schema-less
|
||||
* sql-like query language
|
||||
* support multiple databases with authentication
|
||||
* single time series should scale horizontally (no hot spots)
|
||||
* dynamic cluster changes and data balancing
|
||||
* pubsub layer
|
||||
* continuous queries (keep connection open and return new points as they arrive)
|
||||
* Delete ranges of points from any number of timeseries (that should reflect in disk space usage)
|
||||
* querying should support one or more timeseries (possibly with regex to match on)
|
||||
|
||||
New Requirements
|
||||
----------------
|
||||
* Easy to backup and restore
|
||||
* Large time range queries with one column ?
|
||||
* Optimize for HDD access ?
|
||||
* What are the common use cases that we should optimize for ?
|
||||
|
||||
Modules
|
||||
-------
|
||||
|
||||
|
||||
+--------------------+ +--------------------+
|
||||
| | | |
|
||||
| WebConsole/docs | | Http API |
|
||||
| | | |
|
||||
+------------------+-+ +-+------------------+
|
||||
| |
|
||||
| |
|
||||
+-----+-------+-----------+
|
||||
| |
|
||||
| Lang. Bindings |
|
||||
| |
|
||||
+-----------------+ |
|
||||
| | |
|
||||
| Query Engine | |
|
||||
| | |
|
||||
+-----------------+-------+
|
||||
| |
|
||||
+----+ Coordinator (consensus) +-----+
|
||||
| | | |
|
||||
| +-------------------------+ |
|
||||
| |
|
||||
| |
|
||||
+--------+-----------+ +-------+------------+
|
||||
| | | |
|
||||
| Storage Engine | | Storage Engine |
|
||||
| | | |
|
||||
+--------+-----------+ +-------+------------+
|
||||
|
||||
Replication & Consensus Notes
|
||||
-----------------------------
|
||||
|
||||
Single raft cluster for which machines are in cluster and who owns which locations.
|
||||
1. When a write comes into a server, figure out which machine owns the data, proxy out to that.
|
||||
2. The machine proxies to the server, which assigns a sequence number
|
||||
3. Each machine in the cluster asks the other machines that own hash ring locations what their latest sequence number is every 10 seconds (this is read repair)
|
||||
|
||||
For example, take machines A, B, and C. Say B and C own ring location #2. If a write comes into A it will look up the configuration and pick B or C at random to proxy the write to. Say it goes to B. B assigns a sequence number of 1. It keeps a log for B2 of the writes. It will also keep a log for C2's writes. It then tries to write #1 to C.
|
||||
|
||||
If the write is marked as a quorum write, then B won't return a success to A until the data has been written to both B and C. Every so often both B and C will ask each other what their latest writes are.
|
||||
|
||||
Taking the example further, if we had server D that also owned ring location 2. B would ask C for writes to C2. If C is down it will ask D for writes to C2. This will ensure that if C fails no data will be lost.
|
||||
|
||||
Coding Style
|
||||
------------
|
||||
|
||||
1. Public functions should be at the top of the file, followed by a comment `// private functions` and all private functions.
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
### Packaging
|
||||
|
||||
In order to support all distros and old libc we build and package on
|
||||
centos6.4 (64 bit). The package script takes care of cross compiling
|
||||
Influxdb for 32bit architectures.
|
||||
|
||||
TODO: move the following to chef recipe.
|
||||
|
||||
Below are the steps needed to setup a centos6.4 vm to build Influxdb.
|
||||
|
||||
```
|
||||
sudo yum install git-core glibc.i686 hg bzr protobuf-compiler glibc-devel.i686 libstdc++.i686 libstdc++-devel.i686 python-pip
|
||||
sudo pip install --upgrade awscli
|
||||
git clone git@github.com:influxdb/influxdb.git
|
||||
wget http://go.googlecode.com/files/go1.1.2.linux-amd64.tar.gz
|
||||
mkdir bin
|
||||
cd bin
|
||||
tar -xzvf ../go1.1.2.linux-amd64.tar.gz
|
||||
mv go go1.1.2
|
||||
ln -s go1.1.2 go
|
||||
cd ..
|
||||
curl -L https://get.rvm.io | bash -s stable
|
||||
rvm install 1.9.3
|
||||
cd influxdb/
|
||||
git clean -dfx
|
||||
git checkout .
|
||||
git pull --rebase
|
||||
# PATH=$HOME/bin/go/bin:$PATH GOROOT=~/bin/go/ ./test.sh
|
||||
PATH=$HOME/bin/go/bin:$PATH GOROOT=~/bin/go/ ./release.sh 0.1.1.rc3
|
||||
```
|
||||
Loading…
Reference in New Issue