4224b693d9
refactor: combine preservation.rs and persistence.rs ( #1692 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-11 11:33:14 +00:00
e34d157f28
fix: comments
2021-06-11 07:30:49 -04:00
ef76b350db
Merge pull request #1690 from influxdata/crepererum/inline_parquet_table_struct
...
refactor: inline `Table` into `parquet_file::chunk::Chunk`
2021-06-11 11:26:50 +00:00
71e2a8fbaa
Merge branch 'main' into crepererum/inline_parquet_table_struct
2021-06-11 11:22:48 +00:00
ea9edef716
fix: testing option
2021-06-11 07:18:33 -04:00
fb639ee54f
feat: add UnionExec on top of the scan activities
2021-06-11 07:06:08 -04:00
42d5076772
Merge pull request #1688 from influxdata/fixjaeger
...
fix: Expose jaeger knobs and default max-packet-size to something that works everywhere
2021-06-11 11:02:39 +00:00
95cb5bce8b
Merge branch 'main' into fixjaeger
2021-06-11 10:58:17 +00:00
0cbe74dbde
fix: persistence to parquet by swapping order of arguments ( #1687 )
...
* fix: fix order of arguments
* test: for persistence
2021-06-11 10:55:40 +00:00
f8a518bbed
refactor: inline `Table` into `parquet_file::chunk::Chunk`
...
Note that the resulting size estimations are different because we were
double-counting `Table`. `mem::size_of::<Self>()` is recursive for
non-boxed types since the child will be part of the parent structure.
Issue: #1295 .
2021-06-11 11:54:31 +02:00
5a68abaa53
feat: Implement LayeredTracing
...
__Rationale__
We currently use the `tracing` framework to output to both log outputs (e.g. stdout for k8s) and distributed tracing collectors (e.g. opentelemetry jaeger).
However, due to a limitation in the `tracing` SDK, we can only have one "filter" level that applies
to both logs and tracing outputs. This is unpractical because tracing collectors are designed
to receive high verbosity data (which will be then sampled within the opentelemetry library),
while logs generally are limited to the DEBUG level on production.
This PR adds a `FilteredLayer` tracing subscriber layer, that wraps a subscriber layer with a independent
filter, which can filter events goint to the wrapper subscriber layer more agressively than the global layer.
This will allow us to emit logs at INFO or DEBUG level while passing all events to opentelemetry at TRACE
level (and opentelemetry SDK will then sample the events so that only a small part will be sent to the
ot collector)
__Note__
This PR just implements the `FilteredLayer` and a test. Another PR will integrate this with
our log/tracing setup code.
2021-06-11 04:32:47 +02:00
369c2237f6
fix: Expose jaeger knobs and default max-packet-size to something that works everywhere
...
`--traces-exporter-jaeger-max-packet-size` is important also when you run the jaeger collector
on "localhost" by running `docker run jaegertracing/all-in-one ....` which on mac doesn't really
work on the real localhost but has a few hops between tunneling interfaces, so you'd get mysteriously
dropped packets that can easily drive you to doubt your own sanity on an otherwise calm Thursday evening.
2021-06-11 01:42:26 +02:00
13dd4b23fd
fix: make pruning debug log less confusing ( #1684 )
2021-06-10 18:35:04 +00:00
1c7b13f0a5
Merge pull request #1676 from influxdata/ntran/dedup_merge_exec
...
feat: hook SortPreservingMergeExec into deduplication framework
2021-06-10 17:18:00 +00:00
16b268402e
Merge branch 'main' into ntran/dedup_merge_exec
2021-06-10 17:13:49 +00:00
46d4ab1f2a
refactor: address review comments
2021-06-10 13:13:02 -04:00
11b25b3aaf
refactor: swap order of partition and table in in-memory catalog ( #1678 )
...
* refactor: swap order of partition and table in in-memory catalog
* chore: review feedback
* chore: validate panic message
* chore: review feedback
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-10 16:40:30 +00:00
13bb290a7c
chore: enforce `clippy::future_not_send` for `server` + top-level crate ( #1679 )
...
* chore: enforce `clippy::future_not_send` for `server`
* chore: enforce `clippy::future_not_send` for top-level crate
2021-06-10 15:01:12 +00:00
ab92c0c321
Merge pull request #1668 from influxdata/crepererum/issue1381
...
feat: catalog checkpointing infrastructure
2021-06-10 14:05:46 +00:00
28d1dc4da1
chore: bump preserved catalog version
2021-06-10 16:01:13 +02:00
80ee36cd1a
refactor: slightly streamline path parsing code in pres. catalog
2021-06-10 15:59:28 +02:00
7e7332c9ce
refactor: make comparison a bit less confusing
2021-06-10 15:42:21 +02:00
fd581e2ec9
docs: fix confusion wording in `CatalogState::files`
2021-06-10 15:42:21 +02:00
edbaaedfc3
docs: clarify behavior of unspecified transaction encoding
2021-06-10 15:42:21 +02:00
be9b3a4853
fix: protobuf lint fixes
2021-06-10 15:42:21 +02:00
294c304491
feat: impl catalog checkpointing infrastructure
...
This implements a way to add checkpoints to the preserved catalog and
speed up replay.
Note: This leaves the "hook it up into the actual DB" for a future PR.
Issue: #1381 .
2021-06-10 15:42:21 +02:00
188cacec54
refactor: use `Arc` to pass `ParquetFileMetaData`
...
This will be handy when the catalog state must be able to return
metadata objects so that we can create checkpoints, esp. when we use
multi-chunk parquet files in some midterm future.
2021-06-10 15:42:21 +02:00
c7412740e4
refactor: prepare to read and write multiple file types for catalog
...
Prepares #1381 .
2021-06-10 15:42:21 +02:00
33e364ed78
feat: add encoding info to transaction protobuf
...
This should help with #1381 .
2021-06-10 15:42:21 +02:00
876f642860
test: add benchmark for catalog loading
2021-06-10 15:42:21 +02:00
29dadba4f3
chore: update dependencies ( #1673 )
...
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2021-06-10 13:01:31 +00:00
fd8fbeb89d
Merge pull request #1677 from influxdata/crepererum/clippy_future_not_send_part2
...
chore: enforce `clippy::future_not_send` for `influxdb_iox_client,influxdb2_client,query,tracker`
2021-06-10 12:54:47 +00:00
7bacef6835
chore: enforce `clippy::future_not_send` for `tracker`
2021-06-10 09:52:57 +02:00
7b1106ff64
chore: enforce `clippy::future_not_send` for `query`
2021-06-10 09:48:35 +02:00
3832987ecb
chore: enforce `clippy::future_not_send` for `influxdb2_client`
2021-06-10 09:46:35 +02:00
1c5a779b83
chore: enforce `clippy::future_not_send` for `influxdb_iox_client`
2021-06-10 09:45:02 +02:00
6d3432c702
Merge pull request #1671 from influxdata/crepererum/clippy_future_not_send_part1
...
chore: enforce `clippy::future_not_send` for `parquet_file,object_store`
2021-06-10 07:24:04 +00:00
3ba27bdbd9
Merge branch 'main' into crepererum/clippy_future_not_send_part1
2021-06-10 07:19:31 +00:00
21f6f58e97
Merge pull request #1670 from influxdata/crepererum/extract_server_init
...
refactor: extract server init code
2021-06-10 07:19:04 +00:00
5f863a59fd
Merge branch 'main' into crepererum/extract_server_init
2021-06-10 07:14:57 +00:00
44d8fb9472
Merge branch 'main' into crepererum/clippy_future_not_send_part1
2021-06-10 07:10:11 +00:00
4cf05df35b
feat: hook SortPreservingMergeExec into deduplication framework
2021-06-09 23:29:44 -04:00
feed19ac8b
Merge pull request #1666 from influxdata/ntran/dedup_within_chunk
...
feat: Add Sort operator into the deduplicate plan
2021-06-09 19:13:36 +00:00
4478d900ee
refactor: capture test output
2021-06-09 15:09:13 -04:00
8cc99e3420
Merge branch 'ntran/dedup_within_chunk' of https://github.com/influxdata/influxdb_iox into ntran/dedup_within_chunk
2021-06-09 14:40:29 -04:00
b3c94b9d65
refactor: change order of fields to pass circle CI tests
2021-06-09 14:40:10 -04:00
eed73a30c5
Merge branch 'main' into ntran/dedup_within_chunk
2021-06-09 18:19:17 +00:00
c1c58018fc
refactor: address review comments
2021-06-09 14:17:47 -04:00
89fcc457f4
fix: Fix bug in chunk overlap calculation due to nulls ( #1669 )
...
* fix: Fix bug in chunk overlap calculation due to nulls
* docs: add note about algorithmic complexity
* fix: avoid recursion in normal case
2021-06-09 17:46:39 +00:00
4fe2d7af9c
chore: enforce `clippy::future_not_send` for `parquet_file`
2021-06-09 18:18:27 +02:00
Andrew Lamb
Nga Tran
kodiakhq[bot]
Marco Neumann
Marko Mikulicic
Raphael Taylor-Davies