cf4fd500b9
refactor: remove tests moved to query_tests
2021-09-16 15:05:48 -04:00
472e8a9e49
fix: fix compile error
2021-09-16 15:02:18 -04:00
2bae14df60
test: delete tests
2021-09-16 14:51:26 -04:00
4ae702549f
Merge pull request #2461 from influxdata/jpg/datagen-file-perf
...
perf: Don't open the output file each time we write.
2021-09-16 17:33:45 +00:00
997a96cf30
Merge branch 'main' into jpg/datagen-file-perf
2021-09-16 17:24:37 +00:00
9b3a13f984
feat: remove legacy http list partitions endpoint ( #2557 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 16:56:20 +00:00
f34eab70b3
feat: add pbjson bytes support ( #2560 )
...
* feat: add pbjson bytes support
* chore: fix lint
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 16:46:12 +00:00
315cbb8105
Merge pull request #2561 from influxdata/crepererum/arc_delete_predicates
...
refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates
2021-09-16 15:26:20 +00:00
ec943081c7
refactor: `Arc<Vec<...>>` => `Vec<Arc<...>>` for del predicates
...
The motivations are:
1. The API uses a SINGLE predicate and adds that to many chunks. With
`Arc<Vec<...>>` you gain nothing, with `Vec<Arc<...>>` the predicate
is only stored once (in many vectors)
2. While we currently add predicates blindly to all chunks, we can be way
smarter in the future and prune out tables, partitions or even single
chunks (based on statistics). With that, it will be rare that many
chunks share the exact same set of predicates.
3. It would be nice if we could de-duplicate predicates when writing them
to the preserved catalog without needing to repeat the pruning
discussed in point 2. This is way easier to implement whan chunks
exists in `Arc`s.
4. As a side-note: the `Arc<Vec<...>>` wasn't really cloned around but
instead was created many time. So the new version should be more
memory efficient out of the box.
2021-09-16 17:16:09 +02:00
69939a5ae2
perf: Don't open the output file each time we write.
...
This improves performance of the the file output mode, which should
make it easier to improve the performance of the core generation
logic.
Benchmarked via:
```
time \
./target/release/iox_data_generator \
--spec iox_data_generator/schemas/fully-supported.toml \
--output /tmp/out \
--start '1 month ago'
```
Before:
```
Submitted 271608 total points
real 10.912 10911567us
user 3.129 3129032us
sys 6.257 6257340us
cpu 86%
mem 7152 KiB
```
After:
```
Submitted 271588 total points
real 2.291 2291364us
user 1.969 1969357us
sys 0.058 58030us
cpu 88%
mem 7104 KiB
```
That's 21.0% of the previous time.
2021-09-16 10:57:59 -04:00
ce224bd37f
fix: Capture query execution traces for storage gRPC queries as well ( #2553 )
...
* fix: Capture query execution traces for storage gRPC queries as well
* refactor: remove debugging droppings
* refactor: do not Box::pin within TracedStream
* refactor: Use Futures::TryStreamExt rather than custom collect function
* fix: remove wild println
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 14:45:20 +00:00
ad4b8532bf
Merge pull request #2522 from influxdata/ntran/delete_read
...
feat: push delete predicate down to RUB and OS chunks to eliminate deleted data at query time
2021-09-16 13:31:32 +00:00
33cd1cffad
Merge branch 'main' into ntran/delete_read
2021-09-16 13:22:50 +00:00
b13fb3ea93
Merge pull request #2541 from influxdata/crepererum/time_tests
...
test: even more time-related lifting
2021-09-16 12:05:52 +00:00
fb8a376a04
Merge branch 'main' into crepererum/time_tests
2021-09-16 11:57:42 +00:00
a478138756
refactor: Add SpanContext:new() to make a new span ( #2551 )
...
* refactor: Add SpanContext::new() and remove make_span
* fix: generate random trace_id and span_ids
* docs: Update trace/src/ctx.rs
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-16 11:02:28 +00:00
16157daad8
Merge pull request #2555 from influxdata/er/fix/read_buffer/ord
...
fix: Ensure byte trimmed floats can handle arbitrary float literals
2021-09-16 09:10:13 +00:00
bb17b4e2c2
test: even more time-related lifting
...
Lift a few `Utc::now()` calls further and narrow down checks in tests.
Also avoid a few `<` comparisons which might not always hold.
2021-09-16 11:05:10 +02:00
e51dd0365a
refactor: PR feedback
...
Co-authored-by: Marko Mikulicic <mkm@influxdata.com>
2021-09-16 10:01:44 +01:00
f7228ddd60
test: add test for byte trimmed floats
2021-09-16 10:01:44 +01:00
d387108dab
fix: float byte trimmer filter range
2021-09-16 10:01:44 +01:00
0250bd1337
fix: ensure range filter works with null
2021-09-16 10:01:44 +01:00
1a70865a03
fix: ensure float byte trimmed predicate pushdown works for unencodable values
2021-09-16 10:01:44 +01:00
483508e3c6
feat: add rle method for identifying all non-null row IDs
2021-09-16 10:01:44 +01:00
d04a0d1137
feat: add method for identifying all non-null row IDs
2021-09-16 10:01:44 +01:00
70b0ba44b3
test: failing filter test
2021-09-16 10:01:44 +01:00
ca1973fca7
Merge pull request #2556 from influxdata/crepererum/preserved_catalog_api_split
...
refactor: split preserved catalog `api` into `core` and `interface`
2021-09-16 08:38:37 +00:00
2820db5583
refactor: split preserved catalog `api` into `core` and `interface`
...
This makes it clearer which traits and functions users of the preserved
catalog must implement. This also splits the error types into smaller
enums that are easier to understand.
This change should make it easier to implement new functionality (like
capturing delete predicates).
2021-09-16 10:30:11 +02:00
1d55d9a1b5
feat: add pbjson support ( #2468 )
...
* feat: add pbjson support
* chore: fix test
2021-09-16 07:33:27 +00:00
91fd32d506
fix: Reset restored db's state instead of restarting background worker
2021-09-15 19:04:05 -04:00
61e1eac135
fix: fix the cases of multi[le expressions in delete predicate
2021-09-15 17:00:21 -04:00
d70d94100e
refactor: Extract a type alias
2021-09-15 17:00:09 -04:00
81feced9d6
fix: Restart database background worker when it's restored
2021-09-15 16:59:49 -04:00
7c81c280cf
fix: Shut down a database when it's deleted
2021-09-15 16:59:49 -04:00
7b6d8f9327
feat: Add an API for restoring a database that was marked deleted
2021-09-15 16:59:37 -04:00
185f45f56b
chore: Update datafusion + other deps ( #2545 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 20:56:44 +00:00
74f1e0377c
Merge pull request #2547 from influxdata/cn/more-database-states
...
test: Check for other error Database States
2021-09-15 19:52:51 +00:00
957df68da3
Merge branch 'main' into cn/more-database-states
2021-09-15 19:44:11 +00:00
c66095cad1
feat: remove metrics crate ( #2552 )
2021-09-15 19:43:33 +00:00
6e7fa3e574
feat: migrate http ingest metrics ( #2542 )
...
* feat: migrate http ingest metrics
* chore: review feedback
* refactor: RAII entry ingest recorder
2021-09-15 19:01:10 +00:00
7175488133
chore: add some comments
2021-09-15 14:45:04 -04:00
8f54dbcb58
test: Check for other error Database States
...
Connects to #2539 .
2021-09-15 14:44:14 -04:00
3486cc8b38
fix: should not send an empty delete predicate predicate which means delete everything (no time range)
2021-09-15 14:14:26 -04:00
74d3c2e6d2
feat: Translate DataFusion execution metrics to IOx Spans ( #2529 )
...
* feat: Translate DataFusion execution metrics to IOx Spans
* fix: add end to end test to ensure plumbing is hookedup
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 18:14:23 +00:00
1ea4335ff3
fix: report correct DatabaseStateCode ( #2543 )
...
* fix: report correct DatabaseStateCode
* chore: fix lint
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 18:02:54 +00:00
6f2301e16c
feat: migrate write buffer metrics ( #2536 )
...
* feat: migrate write buffer metrics
* feat: update server/src/write_buffer.rs
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: Andrew Lamb <alamb@influxdata.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-09-15 17:41:55 +00:00
c4d39f328f
Merge pull request #2537 from influxdata/crepererum/issue2493c
...
feat: `Predicate` serialization
2021-09-15 16:27:02 +00:00
72021e010f
test: test unsupported cases of predicate serialization
2021-09-15 18:18:05 +02:00
c5ebf4a2e6
refactor: remove unused predicate serialization variants
2021-09-15 18:05:11 +02:00
44eb3b994d
feat: `Predicate` serialization
...
Closes #2493 .
2021-09-15 16:37:25 +02:00
Nga Tran
kodiakhq[bot]
Raphael Taylor-Davies
Marco Neumann
Jake Goulding
Andrew Lamb
Edd Robinson
Carol (Nichols || Goulding)