Commit Graph

3979 Commits (11ba4b5f6ab56bed9ca46497bb1301ea85551026)

Author SHA1 Message Date
Nga Tran 11ba4b5f6a fix: fix unit_test setting to have the desired results 2021-07-22 14:22:08 -04:00
Nga Tran b2063fb29f test: fix the stats and discover a bug in compaction/split/dedupplication 2021-07-21 17:40:48 -04:00
kodiakhq[bot] 13ae2f0d78
Merge pull request #2070 from influxdata/ntran/dedup_compare_cols_order
feat: new algorithm to compute key ranges for deduplication
2021-07-21 15:50:11 +00:00
kodiakhq[bot] 18dd108ba6
Merge branch 'main' into ntran/dedup_compare_cols_order 2021-07-21 15:42:30 +00:00
Nga Tran 86add39175 refactor: address review comments 2021-07-21 11:41:21 -04:00
kodiakhq[bot] 56dd430d8f
Merge pull request #2077 from influxdata/crepererum/sequencer_metrics
feat: write buffer ingestion metrics
2021-07-21 13:30:22 +00:00
kodiakhq[bot] 91acf3911c
Merge branch 'main' into crepererum/sequencer_metrics 2021-07-21 13:23:23 +00:00
Marco Neumann 55490c279a fix: Kafka watermark error for new partitions 2021-07-21 15:21:52 +02:00
Marco Neumann cddf94653c refactor: use `write_buffer` subsystem for ingest metrics 2021-07-21 15:07:59 +02:00
Marco Neumann fd00206fbb refactor: increase watermark update frequence to once per 10s 2021-07-21 15:02:48 +02:00
Marco Neumann 2f1efcf517 docs: clarify difference 2021-07-21 15:00:53 +02:00
Marco Neumann 4d5f209030 docs: do not repeat unix that often 2021-07-21 14:59:07 +02:00
Marco Neumann 7d597d1d5c refactor: make ingest metrics easier to understand 2021-07-21 13:57:53 +02:00
Raphael Taylor-Davies ffe6e62aee
feat: add instant to datetime conversion (#2078)
* feat: add instant to datetime conversion

* chore: review feedback
2021-07-21 11:43:27 +00:00
Marco Neumann fb931bb1ca feat: write buffer ingestion metrics 2021-07-21 11:59:52 +02:00
Marco Neumann 5df88c70aa feat: add ability to fetch watermarks from write buffer 2021-07-21 11:59:52 +02:00
kodiakhq[bot] 58108b79ec
Merge pull request #2058 from influxdata/pd/add-cache-config
feat: add parquet cache size setting to database rules
2021-07-21 09:42:07 +00:00
kodiakhq[bot] 94a45339fd
Merge branch 'main' into pd/add-cache-config 2021-07-21 09:35:26 +00:00
Andrew Lamb 387667330a
chore: Update datafusion deps (#2073)
* chore: Update datafusion deps

* fix: update tests
2021-07-21 08:27:03 +00:00
Paul Dix a4704dd165 chore: update parquet_cache_limit to u64 and 0 for default 2021-07-20 15:41:06 -04:00
Paul Dix 297e059085 feat: add parquet cache size setting to database rules 2021-07-20 15:41:06 -04:00
Nga Tran d547c22e97 refactor: comments 2021-07-20 15:27:41 -04:00
Nga Tran 150e166813 refactor: fix comments 2021-07-20 15:16:24 -04:00
Nga Tran fa6d216a85 refactor: cleanup 2021-07-20 15:11:02 -04:00
Nga Tran b98888e8d6 feat: implement key_ranges function that uses new range identify algo 2021-07-20 14:58:54 -04:00
Raphael Taylor-Davies 61da0fe4df
fix: update last_instant when rotating into persistable window (#2067) 2021-07-20 16:38:28 +00:00
Raphael Taylor-Davies 091837420f
feat: add PersistenceWindows sytem table (#2030) (#2062)
* feat: add PersistenceWindows sytem table (#2030)

* chore: update log

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 13:10:57 +00:00
Raphael Taylor-Davies e4d2c51e8b
fix: update PersistenceWindows on rules update (#2018) (#2060)
* fix: update PersistenceWindows on rules update (#2018)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 12:44:47 +00:00
kodiakhq[bot] 7d9e1f9704
Merge pull request #2059 from influxdata/crepererum/writer_buffer_seek
feat: implement `seek` for write buffer
2021-07-20 12:36:20 +00:00
kodiakhq[bot] 58dd7e9532
Merge branch 'main' into crepererum/writer_buffer_seek 2021-07-20 12:29:18 +00:00
kodiakhq[bot] 2a7848cbf2
Merge pull request #2064 from influxdata/biggermsg
fix: Increase kafka message size to 30MiB
2021-07-20 12:28:55 +00:00
kodiakhq[bot] a4951b5835
Merge branch 'main' into biggermsg 2021-07-20 12:22:19 +00:00
Raphael Taylor-Davies cf8a60252d
refactor: split system_tables module into smaller modules (#2061)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 12:19:20 +00:00
Marko Mikulicic c01cfbc34c
fix: Increase kafka message size 2021-07-20 14:17:37 +02:00
kodiakhq[bot] cf30d19fd7
Merge pull request #2063 from influxdata/er/fix/flaky_compact_test
test: ensure high enough limit
2021-07-20 12:02:27 +00:00
Marco Neumann ec7ebdff29 refactor: use lifetimes to ensure single stream / no seek while streaming 2021-07-20 13:52:33 +02:00
Edd Robinson cc0aaa58a7 test: ensure high enough limit 2021-07-20 12:43:10 +01:00
Marco Neumann b0663a0337 feat: disallow multiple write buffer streams and seeking while streams
Multiple streams will mess up ordering. Seeking while streaming is
likely a bug and should not work.
2021-07-20 12:35:20 +02:00
Raphael Taylor-Davies 767c2a6fe1
refactor: explicit server startup state machine (#2040)
* refactor: explicit server startup state machine

* chore: update `ServerStage` docs

* chore: further docs

* chore: more logging

* chore: format
2021-07-20 10:11:18 +00:00
Andrew Lamb 2c20528c69
chore: use upstream versions of some workarounds (#2057)
* chore: use upstream versions of some workarounds

* docs: update docstring

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 08:53:46 +00:00
Raphael Taylor-Davies 8e5d5928cf
feat: compute WriteSummary from PersistenceWindows (#2030) (#2054)
* feat: compute WriteSummary from PersistenceWindows (#2030)

* chore: review feedback

Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-20 08:46:52 +00:00
Marco Neumann 38f4eec20e feat: implement `seek` for write buffer
This is required to control replay ranges.
2021-07-20 10:25:56 +02:00
kodiakhq[bot] 07c8dea6b9
Merge pull request #2055 from influxdata/uplimit
feat(iox): Make max http request size configurable
2021-07-20 07:19:56 +00:00
kodiakhq[bot] 6fd40d9e18
Merge branch 'main' into uplimit 2021-07-20 07:11:35 +00:00
Nga Tran 1668420ded feat: new algorth to compute key ranges for deduplicating data 2021-07-19 18:04:25 -04:00
kodiakhq[bot] dde7d19ae9
Merge pull request #2041 from influxdata/jg/snafu-driveby
refactor: Make more use of SNAFU's context methods and ensure! macro
2021-07-19 20:27:29 +00:00
kodiakhq[bot] 5bf68c4a57
Merge branch 'main' into jg/snafu-driveby 2021-07-19 20:20:30 +00:00
Marko Mikulicic 45ff5e214c
feat(iox): Make max http request size configurable 2021-07-19 22:14:51 +02:00
Andrew Lamb 1c16988a51
chore: Update datafusion references (#2056) 2021-07-19 18:09:06 +00:00
Raphael Taylor-Davies 1c8c227668
refactor: push database rules update into Db (#2052)
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-07-19 16:05:21 +00:00