1525069213
Merge pull request #8892 from influxdata/jl-tag-values
...
auth: add series auth to 'show tag values'
2017-10-03 08:47:39 -04:00
7e5fd14e8a
add in some optimization
2017-10-02 12:02:38 -06:00
a6cbce0d3e
fix issues brought up by joe
2017-10-02 11:41:03 -06:00
38dc837910
Fix a minor memory leak when batching points for some services.
...
fixes #8895
2017-10-02 11:26:25 -06:00
1443b22379
auth: add series auth to 'show tag values'
2017-09-27 20:01:18 -04:00
e0cba4477c
Merge pull request #8885 from influxdata/er-entry-race
...
Fix race on Cache entry
2017-09-27 18:42:45 +01:00
d0b81c1e6c
Fix race on Cache entry
2017-09-27 18:10:23 +01:00
a1b67160f6
Use math/bits in encoder
2017-09-26 12:51:08 +01:00
7fed382dbf
Merge pull request #8872 from influxdata/jw-mmap
...
Fix long process stalls
2017-09-25 14:49:35 -06:00
122a74c692
Use synchronous IO for wal and tsm writing
...
The fysncs due to large writes when writing to TSM files and the
WAL can eventually cause large pauses. Since we already buffer
writes, using synchronous IO reduces fsync latency by ensuring
the individiual writes hit disk. This spreads out the latecncy
across multiple writes better.
2017-09-25 12:44:57 -06:00
2def219f09
Refactor Shard to further protect Engine
2017-09-25 17:43:30 +01:00
4a67f92acc
Prevent store from directly accessing Shard's engine
2017-09-25 17:43:01 +01:00
8e9cabbb9c
Fix race in TagValues when reaching into engine
2017-09-25 17:43:01 +01:00
7739ff749a
Ensure engine protected by shard mutex
2017-09-25 17:42:30 +01:00
5774b44a4c
Remove MADV_RANDOM
...
This was inadvertently added when merging the solaris and unix
mmap files. This causes large delays due to major page faults.
2017-09-25 10:25:06 -06:00
ea104596f0
Implement TSI index versioning
...
This commit adds a basic TSI versioning scheme, by adding a Version field
to an index's MANIFEST file.
Existing TSI indexes will not have this field present in their MANIFEST
files, and thus will be deemed incomatible with the current version.
Users with existing TSI indexes will be able to remove them, and convert the
resulting inmem indexes to the current version of a TSI index using the
influx_inspect tooling.
2017-09-22 17:59:39 +01:00
1e345aa7a1
Merge pull request #8856 from influxdata/jw-cache
...
Snapshot compaction improvements
2017-09-22 10:45:54 -06:00
44691847e9
Merge branch 'master' into er-8678-tsi1-where
2017-09-22 16:54:49 +01:00
94aba64b88
Re-use index entries slice when writing TSM index
2017-09-21 12:48:16 -06:00
db204f3eb7
Default concurrent compactions to 50% of available cores
2017-09-21 12:48:11 -06:00
deef0c5649
Fix 32bit alignment
2017-09-20 10:00:20 -06:00
61ca1243c7
Increase index disk writer buffer
2017-09-20 09:05:30 -06:00
796de3dcea
Reduce encoder pool checkout contention
...
With higher cardinalities, the encoder pools where become a bottleneck.
This changes the snapshot compactions ot checkout one encoder of each
type and re-use it while writing the snapshots as opposed to repeatedly
checking it out and in.
2017-09-19 15:27:26 -06:00
391a6288c6
Write parallel snapshot for higher cardinalities
2017-09-19 15:27:26 -06:00
0d52b060df
Skip onFileStoreReplace with tsi
2017-09-19 15:27:25 -06:00
4fe81aeee6
Remove manual Gosched from compactions
...
At higher cardinalities, this dramatically slows down compaction throughput.
2017-09-19 15:27:25 -06:00
31e785d676
Don't deduplicate a single value
2017-09-19 15:27:25 -06:00
2ca9ccee1f
Reset snapshot cache outside of write lock
2017-09-19 15:27:25 -06:00
ddeba2c86b
Split large snapshots and write concurrently
2017-09-19 15:27:25 -06:00
9ee305f6f5
Periodically re-allocate cache store
...
This perioically re-allocates the cache store to avoid memory
fragmentation and gradual slow down of the store after repeated
deletes and inserts into the map.
2017-09-19 15:27:25 -06:00
2885b9b310
Remove entrySizeHints map
...
There is a lot of overhead for calculating the hints for larger
cardinalities. This slows down resetting the partitions in the ring.
2017-09-19 15:27:25 -06:00
4124a8ed97
Simplify cache ring
...
The continuum slice is not needed since the number of partitions
doesn't change. This removes the slice to make the mapping simpler.
2017-09-19 15:27:25 -06:00
ed7bc9d825
fix FindValues panic for empty array
2017-09-19 14:23:32 -07:00
92756ec0ad
Reduce allocations, improve readEntries performance by simplifying loop
...
* callers of ReadEntries and Key API can cache allocated slice
2017-09-19 11:57:10 -07:00
baa05de3f8
add benchmarks
2017-09-19 11:47:48 -07:00
cfc6a1cd9f
implement optimization for Include function
...
```
benchmark old ns/op new ns/op delta
BenchmarkIntegerValues_IncludeNone_1000-8 651 6.69 -98.97%
BenchmarkIntegerValues_IncludeMiddleHalf_1000-8 1131 114 -89.92%
BenchmarkIntegerValues_IncludeFirst_1000-8 638 33.9 -94.69%
BenchmarkIntegerValues_IncludeLast_1000-8 1269 32.2 -97.46%
BenchmarkIntegerValues_IncludeNone_10000-8 7751 6.76 -99.91%
BenchmarkIntegerValues_IncludeMiddleHalf_10000-8 11582 1378 -88.10%
BenchmarkIntegerValues_IncludeFirst_10000-8 7911 43.8 -99.45%
BenchmarkIntegerValues_IncludeLast_10000-8 12442 38.4 -99.69%
```
(cherry picked from commit fb93ad5)
2017-09-19 09:53:28 -07:00
ca40c1ad3c
<type>Values.Exclude function uses binary search and copy builtin
...
```
± benchcmp old.txt new.txt
benchmark old ns/op new ns/op delta
BenchmarkIntegerValues_ExcludeNone_1000-8 1285 7.34 -99.43%
BenchmarkIntegerValues_ExcludeMiddleHalf_1000-8 1258 148 -88.24%
BenchmarkIntegerValues_ExcludeFirst_1000-8 1268 7.51 -99.41%
BenchmarkIntegerValues_ExcludeLast_1000-8 1125 27.7 -97.54%
BenchmarkIntegerValues_ExcludeNone_10000-8 12665 7.31 -99.94%
BenchmarkIntegerValues_ExcludeMiddleHalf_10000-8 12039 976 -91.89%
BenchmarkIntegerValues_ExcludeFirst_10000-8 12663 7.29 -99.94%
BenchmarkIntegerValues_ExcludeLast_10000-8 10990 34.9 -99.68%
```
(cherry picked from commit d7a3c23)
2017-09-19 09:53:26 -07:00
940da04a34
Merge pull request #8829 from influxdata/jw-mmap
...
Release mmap pages when shard is cold
2017-09-18 12:08:37 -06:00
31646aae3a
Release mmap pages when shard is cold
...
This instructs the kernel that it can release memory used by mmap'd
TSM files when they are not actively being used. It the mappings are
use, the kernel will fault the pages back in. On linux, this causes
RES memory to drop immediately when run.
2017-09-18 11:51:51 -06:00
e39de3e427
Merge pull request #8782 from oiooj/pr-shard-fix
...
Correctly check if the Shard is ready for queries or writes
2017-09-18 18:17:19 +01:00
2228b91b0d
Unsigned data type parsing and prioritization
2017-09-14 12:28:13 -05:00
7d467c2047
Fix windows unmapping of anonymous index slice
2017-09-12 10:30:10 -06:00
b4b3c159cc
Fixup rebase
2017-09-11 17:04:10 -06:00
d5d9f9acfe
Remove debug line
2017-09-11 15:31:28 -06:00
26f92ce6ac
Remove commented out code
2017-09-11 15:30:05 -06:00
820856347c
Don't use disk temp file for snapshots
2017-09-11 15:29:26 -06:00
4ed9c75896
Fix unmapping anonymous memory slice
2017-09-11 15:29:26 -06:00
97f7857715
Remove mutex on TSMWriter
...
This isn't used by more than one goroutine so locks are unnecessary.
2017-09-11 15:29:26 -06:00
a93a5e9bdf
Include the size of the key in the cache size
2017-09-11 15:29:26 -06:00
38460ec37e
Re-enable compactions during writes
...
A cold shard that suddenly receives a lot of writes could get a very
big cache that takes a long time to snapshot or causes the cache
max memory limit to be hit more quickly. This re-enables the compactions
if necessary during writes so we don't have to wait for the shard monitor
goroutine to re-enable them.
2017-09-11 15:29:26 -06:00
Joe LeGasse
Lyon Hill
Edd Robinson
Jason Wilder
Jason Wilder
Stuart Carnie
Jonathan A. Sternberg