fe960b0f3a
Add benchmarks for bloom filter
2017-09-22 17:59:32 +01:00
db204f3eb7
Default concurrent compactions to 50% of available cores
2017-09-21 12:48:11 -06:00
a40b2bb210
Simplify bloom filter invalidation.
2017-09-11 15:29:26 -06:00
d3e832b462
Use offheap memory for indirect index offsets slice
2017-09-11 15:29:25 -06:00
4009223fb6
Avoid allocating murmur3.Hash too frequently
...
These hashes were getting allocate very frequently with high cardinality
datasets. This allows them to be re-used.
2017-09-11 15:26:24 -06:00
e43bec4a3a
Test slices strings Exists* functions
2017-08-08 20:33:26 -04:00
a43238618e
Merge pull request #8512 from axiomhq/loglogbeta
...
Switch to LogLog-Beta Cardinality estimation
2017-07-07 16:14:16 +01:00
4cb01c1768
change beta constants for the hll cardinality bias estimator
2017-06-30 07:47:16 -07:00
cf6d4b6e71
Fix building on solaris/illumos
2017-06-21 09:16:45 -06:00
643b2eb30c
Switch to LogLog-Beta Cardinality estimation
...
The new algorithm uses only one formula and needs no additional bias corrections for the entire range of cardinalities,
therefore, it is more efficient and simpler to implement. Our simulations show that the accuracy provided by the new
algorithm is as good as or better than the accuracy provided by either of HyperLogLog or HyperLogLog++. The sparse
representation was kept in to provide better low cardinality accuracy. However the linear counting and range estimations
are replaced.
2017-06-20 15:25:01 +02:00
1975940f76
intermediate compaction commit
2017-05-23 08:42:25 -06:00
48a06432df
Add tsi1 bloom filter.
2017-05-23 08:41:31 -06:00
623ff67221
Fix HLL variableLengthList size decoding.
2017-05-19 11:44:25 -06:00
503d41a08f
Add LimitedBytePool for wal buffers
...
This pool was previously a pool.Bytes to avoid repetitive allocations.
It was recently switchted to a sync.Pool because pool.Bytes held onto
very larger buffers at times which were never released. sync.Pool is
showing up in allocation profiles quite frequently.
This switches the pool to a new pool that limits how many buffers are
in the pool as well as the max size of each buffer in the pool. This
provides better bounds on allocations.
2017-05-11 11:27:00 -06:00
0d74497abe
Reset rhh map elements to reuse allocations.
2017-04-04 11:57:37 -06:00
95d4016ff2
Merge branch 'tsi' of https://github.com/influxdata/influxdb into tsi-series-block-partitioning
2017-04-04 10:14:03 -06:00
bf49b176f5
Partition tsi1 series index.
2017-04-04 09:46:04 -06:00
8da84e6144
Merge branch 'master' into tsi
2017-04-03 11:21:02 -06:00
1c4ecb12c1
Don't panic on nil engine
2017-03-22 10:07:29 -06:00
1e9fa7bc2c
Fix 32-bit rhh implementation.
2017-03-21 11:44:13 -06:00
70efc70abe
Reduce lock contention, fix rhh lookup.
2017-03-17 09:44:11 -06:00
0ebbfb8f77
hll: skip recalc of sparseSet if tmpSet is empty
...
```
benchmark old ns/op new ns/op delta
BenchmarkSet_Count/set_size_1000-8 38095 28.3 -99.93%
BenchmarkSet_Count/set_size_5000-8 152052 30.1 -99.98%
BenchmarkSet_Count/set_size_10000-8 50953 54978 +7.90%
BenchmarkSet_Count/set_size_50000-8 32495 31222 -3.92%
BenchmarkSet_Count/set_size_1000000-8 32603 30800 -5.53%
benchmark old allocs new allocs delta
BenchmarkSet_Count/set_size_1000-8 4 0 -100.00%
BenchmarkSet_Count/set_size_5000-8 4 0 -100.00%
BenchmarkSet_Count/set_size_10000-8 0 0 +0.00%
BenchmarkSet_Count/set_size_50000-8 0 0 +0.00%
BenchmarkSet_Count/set_size_1000000-8 0 0 +0.00%
benchmark old bytes new bytes delta
BenchmarkSet_Count/set_size_1000-8 16496 0 -100.00%
BenchmarkSet_Count/set_size_5000-8 16497 0 -100.00%
BenchmarkSet_Count/set_size_10000-8 0 0 +0.00%
BenchmarkSet_Count/set_size_50000-8 0 0 +0.00%
BenchmarkSet_Count/set_size_1000000-8 0 0 +0.00%
```
2017-01-31 08:51:05 -07:00
ab94c1b743
Fixes #7882
2017-01-30 19:12:24 +00:00
695adafc00
Add measurement sketches
2017-01-05 10:15:37 -07:00
1339c7b146
Initialise HLL with error
2017-01-05 10:15:37 -07:00
9f8b206b51
Fix measurement system queries.
2017-01-05 10:15:34 -07:00
fa7239ecad
Fix windows mmap
2017-01-05 10:11:12 -07:00
cb93f10120
Remove per-shard in-memory index.
2017-01-05 10:11:09 -07:00
8863e3c0f3
Refactor tsi1 merge iterators, finish multi-file compaction.
2017-01-05 10:01:25 -07:00
e2c3b52ca4
Adds a custom HyperLogLog++ implementation
2017-01-05 10:00:14 -07:00
149b1cef1d
Fix 32bit overflow; limit capacity
2017-01-05 09:59:10 -07:00
2a5c865b44
Use xxhash
2017-01-05 09:57:35 -07:00
bd8dd9a291
Sketches working
2017-01-05 09:54:04 -07:00
d19fbf5ab4
Wire in HLL estimator
2017-01-05 09:54:03 -07:00
57d0556174
Fix 32-bit issues.
2017-01-05 09:34:37 -07:00
8d40ceb00c
TSI1 Index
2017-01-05 09:34:36 -07:00
3240af07e0
Fix RHH packing.
2017-01-05 09:34:36 -07:00
e25d61e4bd
TagSet writer & reader.
2017-01-05 09:34:36 -07:00
2c34b24f5c
Implemented SeriesList
2017-01-05 09:34:36 -07:00
6523675c20
Implemented RHH hash map.
2017-01-05 09:34:35 -07:00
0551d87ddb
Update godoc for pkg
2016-12-30 18:03:01 -08:00
fa2b78d67e
Optimize string escaping & unescaping
...
benchmarks pre-commit:
go test ./pkg/escape/... -bench=. -benchmem
BenchmarkStringEscapeNoEscapes-8 10000000 122 ns/op 0 B/op 0 allocs/op
BenchmarkStringUnescapeNoEscapes-8 300000000 5.69 ns/op 0 B/op 0 allocs/op
BenchmarkManyStringEscape-8 1000000 1073 ns/op 320 B/op 8 allocs/op
BenchmarkManyStringUnescape-8 1000000 1557 ns/op 288 B/op 8 allocs/op
PASS
ok github.com/influxdata/influxdb/pkg/escape 6.304s
benchmarks post-commit:
go test ./pkg/escape/... -bench=. -benchmem
BenchmarkStringEscapeNoEscapes-8 100000000 17.5 ns/op 0 B/op 0 allocs/op
BenchmarkStringUnescapeNoEscapes-8 300000000 5.70 ns/op 0 B/op 0 allocs/op
BenchmarkManyStringEscape-8 3000000 451 ns/op 224 B/op 6 allocs/op
BenchmarkManyStringUnescape-8 2000000 944 ns/op 304 B/op 9 allocs/op
PASS
ok github.com/influxdata/influxdb/pkg/escape 8.727s
2016-11-24 10:25:35 +00:00
3fe0ffd5ad
String escape unit tests
2016-11-18 16:02:52 +00:00
743946fafb
models: Add FieldIterator type
...
The FieldIterator is used to scan over the fields of a point, providing
information, and delaying parsing/decoding the value until it is needed.
This change uses this new type to avoid the allocation of a map for the
fields which is then thrown away as soon as the points get converted
into columns within the datastore.
2016-10-03 16:30:21 -06:00
f632b41f6a
Update godoc
2016-09-26 12:19:15 -06:00
7f96d78b79
Make encoder re-usable
...
This allows encoders to be re-used and maintained in a pool to
avoid allocating new ones on every compactions and write of an encoded
block. The pool used is not a sync.Pool to ensure that the encoders
will not be garbage collected.
2016-09-26 12:19:15 -06:00
25508fa05f
Add pool.Bytes for re-using byte slices
...
This adds a new pool type that allows byte slices
to be re-used across clients. A sync.Pool can't be used in some
cases because the the slices not in use end up getting garbage
collected due to frequent GCs.
2016-09-26 12:19:15 -06:00
b692ef4f48
Rename throttle package to limiter
2016-07-18 12:00:58 -06:00
c2370b437b
Limit in-flight wal writes/encodings
...
A slower disk can can cause excessive allocations to occur when
writing to the WAL because the slower encoding and compression occurs
before taking the write lock. The encoding/compression grabs a large
byte slice from a pool and ultimately waits until it can acquire the
write lock.
This adds a throttle to limit how many inflight WAL writes can be queued
up to prevent OOMing the processess with slower disks and heavy writes.
2016-07-17 23:53:12 -06:00
84c776bf2a
Skip allocating string in unescapeString
2016-05-27 10:30:08 -06:00
4b32760dac
Use stdlib func instead of scanning slices
2016-04-20 16:08:58 -06:00
fde5576d64
Avoid allocations when no escape chars present
2016-04-20 16:00:26 -06:00
287b7173db
include missing license for equal.go
2016-04-06 17:08:13 -04:00
d9a6a7340f
add canonical paths
2016-02-10 11:30:52 -07:00
3dd6aa17f3
Test the merge iterator for every type instead of just floats
2016-02-10 09:40:26 -07:00
cde973f409
refactor query engine
2016-02-10 09:40:24 -07:00
a8c285d265
Correctly name pkg README
...
[ci skip]
2015-12-17 11:24:58 -08:00
d36a610a06
Complete lint for pkg/slices
2015-11-09 20:21:03 +01:00
7491564f9d
Added tests for esape.Unescape
2015-10-16 10:14:47 +11:00
7284e314e5
add pkg files
2015-09-16 16:41:56 -05:00
bdc54cf5d2
helpers -> pkg + readme
2015-09-04 13:30:43 -05:00
Edd Robinson
Jason Wilder
Ben Johnson
Matt McCoy
Seif Lotfy
Stuart Carnie
Mark Rushakoff
Cameron Sparr
Joe LeGasse
aviau
Jonathan A. Sternberg
Philip O'Toole
Steffen Windoffer
David Howden
Cory LaNou