4204127b05
refactor: use protobuf for in-parquet metadata
2021-06-30 16:51:37 +02:00
9448b8044e
Merge pull request #1850 from influxdata/crepererum/safer_min_max_sequence
...
refactor: make `MinMaxSequence` safer to use
2021-06-30 14:45:41 +00:00
a2246fd890
Merge branch 'main' into crepererum/safer_min_max_sequence
2021-06-30 14:38:37 +00:00
043890369f
refactor: make `MinMaxSequence` safer to use
2021-06-30 16:37:48 +02:00
635e4d0a7d
refactor: plumbing to prevent compaction starving persistence ( #1852 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 13:43:25 +00:00
530b5cdf22
feat: namespace issues in git-log-prs ( #1849 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 13:35:34 +00:00
b86fb88f60
docs: Edit Quick Start ( #1839 )
...
* docs: start editing Quick Start
* docs: edit Quick Start in README
* fix: reorder Quick start steps
* fix: Edit git instructions
* fix: s/values/options
* fix: Add a line
* fix: Edit release mode section
* fix: note that docker is optional
* fix: fix line break
* fix: fix link
* fix: more edits
* fix: s/Quick Start/Get started
* fix: rename API compatibility section
* fix: fix link
* fix: Apply code review suggestions
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* fix: fix link path
* fix: remove rust version check instruction
Co-authored-by: pierwill <pierwill@users.noreply.github.com>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-30 13:27:49 +00:00
89757d7232
fix: do not print test output to logs except on failure ( #1840 )
...
* fix: do not print test output to logs except on failure
* docs: update CONTRIBUTING.md
2021-06-30 13:20:11 +00:00
403f7297e1
Merge pull request #1844 from influxdata/crepererum/no_catalog_on_db_creation
...
feat: do NOT load preserved catalogs on late DB creation
2021-06-30 10:11:17 +00:00
983062f6fa
Merge branch 'main' into crepererum/no_catalog_on_db_creation
2021-06-30 10:04:00 +00:00
438b65cb94
Merge pull request #1845 from influxdata/er/refactor/read_buffer/table_names
...
refactor: remove table name from Read Buffer API.
2021-06-30 09:50:23 +00:00
e5cf08d05c
Merge branch 'main' into er/refactor/read_buffer/table_names
2021-06-30 09:43:41 +00:00
d52807eb2b
Merge branch 'main' into er/refactor/read_buffer/table_names
2021-06-30 10:43:39 +01:00
62d3305923
feat: optimize the dictionaries in the output of deduplicate node ( #1827 ) ( #1832 )
...
* feat: optimize dedup dictionaries (#1827 )
* fix: handle sliced null bitmasks
* chore: review feedback
2021-06-30 09:30:16 +00:00
2e430ac7f0
refactor: remove table name from read_filter schema
2021-06-30 09:50:53 +01:00
62f274cc1b
refactor: remove table name from column_values
2021-06-30 09:46:54 +01:00
5737c9d962
refactor: remove table name from column_names
2021-06-30 09:43:41 +01:00
c4e054f909
feat: do NOT load preserved catalogs on late DB creation
...
When a DB is created AFTER the server is initialized, then we can assume
it is a new DB (because the rules file did not exist beforehand). We
shall treat it as a new DB with no data and should not try to load some
leftover / stale / whatever preserved catalog for it. How this catalog
came into existence we do not know and it was certainly not properly
managed by IOx. So we error if there is a catalog.
Furthermore the old implementation was kinda broken since it loaded the
perserved catalog "in-sync" with the gRPC call that issued the DB
creation (we only have a delayed init concept for DBs that are loaded on
instance startup). In production that would very likely provoke nasty
timeouts.
On top of that this new behavior will also be somewhat more sane when we
think about sequencer (e.g. Kafka) replays. We certainly do not wanna do
any replays for newly created DBs.
TLDR: New behavior for DBs created via gRPC is "new empty DB". This does
NOT affect DBs loaded on instance startup (aka existing DBs).
2021-06-30 10:12:38 +02:00
58310abfee
refactor: de-duplicate code in `server::db::load`
2021-06-30 10:08:25 +02:00
e6a4e0d709
refactor: make the code clearer for schema even though they are the same
2021-06-29 17:46:30 -04:00
a249b90952
refactor: refactor and add temp info for debugging
2021-06-29 16:35:50 -04:00
4611e5d584
chore: merge main to branch
2021-06-29 15:39:23 -04:00
388e7b7650
fix: reset last_batch
2021-06-29 15:15:09 -04:00
8f309eb569
feat: improve deduplicate to avoid as many concat_batches as possible
2021-06-29 14:41:54 -04:00
fef160e24f
feat: Implement data driven query_tests and port explain tests ( #1814 )
...
* feat: Implment data driven query testing and port explain tests
* fix: do not fmt the auto generated cases
* refactor: split setup and parser into separate modules
* refactor: Add log to runner, add end to end tests
* docs: fixu cpmments
2021-06-29 16:09:51 +00:00
2deda5b14d
Merge pull request #1795 from influxdata/crepererum/mb_checkpoint
...
feat: implement persistence window checkpointing
2021-06-29 15:33:36 +00:00
e10173124f
Merge branch 'main' into crepererum/mb_checkpoint
2021-06-29 15:25:24 +00:00
eac9261507
chore: print end-to-end output ( #1838 )
...
* chore: print end-to-end output
* chore: clippy
* chore: update CONTRIBUTING.md
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 15:09:43 +00:00
0f8ce926f7
Merge pull request #1836 from influxdata/crepererum/issue1821
...
refactor: write parquet files w/o holding the transaction lock
2021-06-29 13:56:16 +00:00
9d10ac9f6a
refactor: write parquet files w/o holding the transaction lock
...
This allows to prepare writes per-tableXpartition before entering the
database-exclusive section that deals with catalog transactions.
Closes #1821 .
2021-06-29 14:23:06 +02:00
ddc9cd49ca
chore: bump preserved catalog version
2021-06-29 14:23:06 +02:00
3ebb6a3037
refactor: do not capture txn-specific information in parquet files
...
This helps with #1821 .
2021-06-29 14:22:36 +02:00
3ae8ac2467
chore: improve wait_for_chunk failure output ( #1835 )
2021-06-29 11:54:32 +00:00
dc96f8f9a8
Merge pull request #1834 from influxdata/er/refactor/table_names
...
refactor: remove table_names from Read Buffer
2021-06-29 11:23:12 +00:00
4bab274bc1
Merge branch 'main' into er/refactor/table_names
2021-06-29 12:15:41 +01:00
5287f6a577
feat: print operations on wait_for_chunk failure ( #1809 ) ( #1833 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 11:09:11 +00:00
a7198ea78b
refactor: use satisfies_predicate in apply_predicate
2021-06-29 11:58:28 +01:00
86e2fe4138
refactor: satisfies predicate
2021-06-29 11:58:28 +01:00
fab04acef5
Merge pull request #1824 from influxdata/crepererum/issue1821-cleanup-lock
...
refactor: improve locking around the parquet file cleanup
2021-06-29 10:55:04 +00:00
eda9532eb2
Merge branch 'main' into crepererum/issue1821-cleanup-lock
2021-06-29 10:48:43 +00:00
48df13de05
refactor: use parking lot for catalog cleanup
2021-06-29 12:47:29 +02:00
3ee96c4618
fix: Do not sequence local writes (avoid panic under load) ( #1826 )
...
* fix: Do not sequence local writes
* fix: Update server/src/db.rs
Co-authored-by: Edd Robinson <me@edd.io>
* fix: review comments
* fix: restore passing sequence information down to mutable buffer
* fix: store min/max times even when there are no sequence numbers
Co-authored-by: Edd Robinson <me@edd.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 10:39:37 +00:00
f824f235b4
fix: fix info log message
...
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
2021-06-29 12:35:05 +02:00
77bd59e308
Merge pull request #1823 from influxdata/crepererum/issue1821-rebuild
...
refactor: only rebuild a single txn for pres. catalogs
2021-06-29 10:06:39 +00:00
144ac8932e
Merge branch 'main' into crepererum/issue1821-rebuild
2021-06-29 09:58:56 +00:00
778a611fb8
docs: add clarifying comment for rebuild test
2021-06-29 11:58:19 +02:00
17f89ea8d0
docs: fix comment about lock downgrade
2021-06-29 11:53:55 +02:00
f145c5d760
feat: add ability to optimize record batch dictionaries ( #1827 ) ( #1830 )
...
* feat: add ability to optimize record batch dictionaries (#1827 )
* chore: correct capacity
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-29 09:23:56 +00:00
2cd5ce98be
refactor: do not pass locks around for catalog cleanup
2021-06-29 10:21:41 +02:00
730a23faa3
refactor: improve locking around the parquet file cleanup
...
Instead of (ab)using the transaction lock to prevent the cleanup job
from removing just-written parquet files, use a dedicated lock. This
will later allow us to write parquet files before starting a transaction
(i.e. w/o holding the transaction lock).
This will help with #1821 .
2021-06-29 10:20:03 +02:00
Marco Neumann
kodiakhq[bot]
Raphael Taylor-Davies
pierwill
Andrew Lamb
Edd Robinson
Nga Tran