Commit Graph

17 Commits (bcc1ad0d3f7b35fff1371337ebf916a8d6aae0fa)

Author SHA1 Message Date
Jason Wilder 6592615958 Updated compaction strategy
This changes compacting files to merge sequences of files in lower generations
up to later generations
2015-12-04 23:30:39 -07:00
Jason Wilder 357b88c439 Increment sequence of max generation when compaction files 2015-12-04 13:46:28 -07:00
Jason Wilder 52bec1f7f6 Change TSM file naming to generation-sequence.tsm 2015-12-04 11:51:33 -07:00
Paul Dix bf65e967aa Add test for compacting multiple TSM files 2015-12-03 10:36:17 -05:00
Paul Dix b0fb8a0a27 Update TSM cache, compact, wal, encoding
* Update cache to have a single slice of values for a key (removed checkpoints)
* Changed compact.Plan to only worry about TSM files.
* Updated Plan to not return an error since there was no case in which it would.
* Update WAL to not keep stats since they're no longer needed.
* Update engine to flush the Cache/WAL to a new TSM file when the min threshold is hit.
* Split compact logic between TSM compacts and WAL/Cache writes.
* Remove unnecessary merge iterator, wal segment iterator, and other no longer necessary stuff.
* Remove the asending bool from the Dedupe method. Values should always be in ascending order. It's up to the cursor to iterate through values based on the direction. Giving the cursor responsibility makes it so we don't need to sort, dedupe or reallocate anything for different query orders.
* Updated engine to use its locks to ensure writes and cache flushes don't cause a race.
* Update all tests with new signatures. Removed a bunch of tests around TSM rewrites and WAL segment iteration that are no longer necessary.
2015-12-03 08:11:50 -05:00
Jason Wilder 751d1dd467 Don't rewrite TSM files while WAL segments exist
This approach is not working and needs to be reworked.
2015-12-02 09:45:24 -07:00
Jason Wilder 5744f5ba02 Add ability to filter values by time when writing TSM files 2015-12-02 09:45:24 -07:00
Jason Wilder 708266da69 Cache related compaction fixes 2015-12-02 09:45:24 -07:00
Jason Wilder 231c052003 Don't limit WAL segments during compaction
Since they are already loaded in the cache, this limit is not really
needed anymore.
2015-12-02 09:45:24 -07:00
Jason Wilder 7e249e0555 Use CacheKeyIterator instead of WALKeyIterator during compactions 2015-12-02 09:45:24 -07:00
Jason Wilder 4a03469662 Integrate TSM compaction into dev engine 2015-12-02 09:45:23 -07:00
Jason Wilder d4b1c25f8e Add CompactionPlanner type
CompactionPlanner is used to determine which files (WAL Segments, TSM
Files) to include in a given compaction run.
2015-12-02 09:45:23 -07:00
Jason Wilder 5291fbcf39 Add TSM support to MergeIterator
Enables the ability to combine multiple TSM files into one as well
as merge existing TSM files with newer WAL segment values.
2015-12-02 09:45:23 -07:00
Jason Wilder acdb6bcdf6 Add TSMKeyIterator
Allows iterating of multiple TSM files in sort key and values order.
2015-12-02 09:45:23 -07:00
Jason Wilder 25206c729c Add compactor type 2015-11-24 08:50:07 -07:00
Jason Wilder f70323cb89 Add MergeIterator
MergeIterator will be used to merge multiple TSM KeyIterators and the
WAL KeyIterator using a stream based iteration approach.  Each iteration
cycle returns a key and values ordered in way to write a new TSM file
optimally.
2015-11-23 14:59:15 -07:00
Jason Wilder 5334271e26 Add KeyIterator for WAL segments
This provides and interface and type to combine multiple WAL segments
in order and then allow the values to be read in an order suitable for
writing to a TSM file.
2015-11-23 14:59:15 -07:00