0f8ce926f7
Merge pull request #1836 from influxdata/crepererum/issue1821
...
refactor: write parquet files w/o holding the transaction lock
2021-06-29 13:56:16 +00:00
9d10ac9f6a
refactor: write parquet files w/o holding the transaction lock
...
This allows to prepare writes per-tableXpartition before entering the
database-exclusive section that deals with catalog transactions.
Closes #1821 .
2021-06-29 14:23:06 +02:00
ddc9cd49ca
chore: bump preserved catalog version
2021-06-29 14:23:06 +02:00
3ebb6a3037
refactor: do not capture txn-specific information in parquet files
...
This helps with #1821 .
2021-06-29 14:22:36 +02:00
3ae8ac2467
chore: improve wait_for_chunk failure output ( #1835 )
2021-06-29 11:54:32 +00:00
dc96f8f9a8
Merge pull request #1834 from influxdata/er/refactor/table_names
...
refactor: remove table_names from Read Buffer
2021-06-29 11:23:12 +00:00
4bab274bc1
Merge branch 'main' into er/refactor/table_names
2021-06-29 12:15:41 +01:00
5287f6a577
feat: print operations on wait_for_chunk failure ( #1809 ) ( #1833 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 11:09:11 +00:00
a7198ea78b
refactor: use satisfies_predicate in apply_predicate
2021-06-29 11:58:28 +01:00
86e2fe4138
refactor: satisfies predicate
2021-06-29 11:58:28 +01:00
fab04acef5
Merge pull request #1824 from influxdata/crepererum/issue1821-cleanup-lock
...
refactor: improve locking around the parquet file cleanup
2021-06-29 10:55:04 +00:00
eda9532eb2
Merge branch 'main' into crepererum/issue1821-cleanup-lock
2021-06-29 10:48:43 +00:00
48df13de05
refactor: use parking lot for catalog cleanup
2021-06-29 12:47:29 +02:00
3ee96c4618
fix: Do not sequence local writes (avoid panic under load) ( #1826 )
...
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
f824f235b4
fix: fix info log message
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-06-29 12:35:05 +02:00
77bd59e308
Merge pull request #1823 from influxdata/crepererum/issue1821-rebuild
...
refactor: only rebuild a single txn for pres. catalogs
2021-06-29 10:06:39 +00:00
144ac8932e
Merge branch 'main' into crepererum/issue1821-rebuild
2021-06-29 09:58:56 +00:00
778a611fb8
docs: add clarifying comment for rebuild test
2021-06-29 11:58:19 +02:00
17f89ea8d0
docs: fix comment about lock downgrade
2021-06-29 11:53:55 +02:00
f145c5d760
feat: add ability to optimize record batch dictionaries ( #1827 ) ( #1830 )
...
* feat: add ability to optimize record batch dictionaries (#1827 )
* chore: correct capacity
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 09:23:56 +00:00
2cd5ce98be
refactor: do not pass locks around for catalog cleanup
2021-06-29 10:21:41 +02:00
730a23faa3
refactor: improve locking around the parquet file cleanup
...
Instead of (ab)using the transaction lock to prevent the cleanup job
from removing just-written parquet files, use a dedicated lock. This
will later allow us to write parquet files before starting a transaction
(i.e. w/o holding the transaction lock).
This will help with #1821 .
2021-06-29 10:20:03 +02:00
5cc773ad80
chore: update deps to get new arrow ( #1831 )
2021-06-28 20:31:31 +00:00
affd74d694
Merge pull request #1829 from influxdata/er/refactor/table_names
...
refactor: clarify intent of applying predicate to Chunk
2021-06-28 17:03:00 +00:00
988fe9abfa
Merge branch 'main' into er/refactor/table_names
2021-06-28 16:55:41 +00:00
6aa5cfa937
Merge pull request #1796 from influxdata/layered-tracing
...
fix: Use layered tracing
2021-06-28 16:45:55 +00:00
12ae9b012a
refactor: clarify intent of
2021-06-28 17:39:48 +01:00
f2c7454274
fix: Use layered tracing
2021-06-28 18:37:12 +02:00
32c683ddc1
fix: remove shared dictionary concept from ChunkSnapshot ( #1570 ) ( #1820 ) ( #1825 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-28 15:26:49 +00:00
9456ae5d39
Merge pull request #1798 from influxdata/cn/fix-loop-problem
...
fix: Try all partition_writes even if one in the middle fails
2021-06-28 13:53:39 +00:00
0f7c47d10e
fix: Limit the number of errors per sequenced entry we'll collect
2021-06-28 09:29:17 -04:00
1e171e2e9a
refactor: Organize `use` statements and let rustfmt manage order
2021-06-28 09:29:15 -04:00
f3a3a9b267
fix: Try to write all partition_writes even if one fails, collect all errors and report at the end
2021-06-28 09:24:23 -04:00
4d2954ec1d
test: Write a failing tests for partition_writes being ignored after a failure
2021-06-28 09:24:23 -04:00
6ec24353bf
refactor: only rebuild a single txn for pres. catalogs
...
Stop relying on in-parquet transaction information during catalog
rebuilds. This has some downsides (no fork detection, only a single
transaction hence no time travel) but will allow that we remove
transaction information from parquet files, so that we can finally move
the actual parquet file storage out of the transaction lock.
This will help with #1821 .
2021-06-28 15:10:44 +02:00
4588c282e1
Merge pull request #1813 from influxdata/fixnewline
...
fix: Fix trailing newline emission in case of truncated log line
2021-06-28 11:20:33 +00:00
b69711d5ab
Merge branch 'main' into fixnewline
2021-06-28 11:13:41 +00:00
4053dcaa2e
Merge pull request #1816 from influxdata/crepererum/move_catalog_loading_code
...
refactor: move catalog loading code into its own module
2021-06-28 11:12:38 +00:00
8927491f04
Merge branch 'main' into crepererum/move_catalog_loading_code
2021-06-28 11:05:59 +00:00
dacc68f8d6
Merge pull request #1819 from influxdata/crepererum/editorconfig_indent
...
chore: add indent to editorconfig
2021-06-28 11:05:05 +00:00
9690d5d3f8
chore: add indent to editorconfig
2021-06-28 12:52:03 +02:00
65e65412cc
refactor: move catalog loading code into its own module
2021-06-28 12:46:25 +02:00
1ff0e8d8d4
fix: Fix trailing newline emission in case of truncated log line
2021-06-28 12:23:53 +02:00
f69d61c7d1
Merge pull request #1801 from influxdata/pd-write-persistence-windows
...
feat: add persistence windows to partition and update on write
2021-06-25 19:52:15 +00:00
af93bb6573
chore: PR cleanup
2021-06-25 15:45:00 -04:00
de236c5a6f
feat: update persistence windows to support late arrival less than 30 seconds
2021-06-25 15:34:11 -04:00
435b4b6a94
feat: add persistence windows to partition and update on write
...
This brings the persistence windows into the catalog partition. It adds a helper method on TableBatch to get the min and max times for a given write. Finally, it adds this logic to the db to update persistence windows on every write while the partition write lock is being held.
2021-06-25 15:34:11 -04:00
9cfabfd1b7
Merge pull request #1810 from influxdata/er/refactor/logging
...
refactor: logging
2021-06-25 16:46:27 +00:00
e033b52bcd
Merge branch 'main' into er/refactor/logging
2021-06-25 16:39:11 +00:00
ed98812c2a
refactor: logging
2021-06-25 17:37:04 +01:00
kodiakhq[bot]
Marco Neumann
Raphael Taylor-Davies
Edd Robinson
Andrew Lamb
Marko Mikulicic
Carol (Nichols || Goulding)
Paul Dix