2021-09-30 08:46:23 +00:00
|
|
|
# Distributed Tracing in IOx
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
Not to be confused with the Rust [tracing](https://docs.rs/tracing) ecosystem, which we use for [logging](logging.md),
|
|
|
|
|
distributed tracing refers to ability to capture information about where time is spent during a given transaction,
|
|
|
|
|
potentially across service boundaries
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
## Components
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Trace (trace)
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
The `trace` crate contains the data model for distributed tracing and nothing else. It is an intentionally lightweight
|
|
|
|
|
dependency, as it is needed by any code that wishes to produce trace spans.
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Trace HTTP (trace_http)
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
The `trace_http` crate contains the logic to extract context information from HTTP requests, and use this to produce
|
|
|
|
|
Spans. This takes the form of a tower [layer] called `TraceLayer` that attaches the `SpanContext` of this generated
|
|
|
|
|
newly Span to the [Request] as an [extension]. This can then be accessed downstream and used to create new child spans.
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
```rust
|
|
|
|
|
async fn handle_request(req: Request<Body>) {
|
|
|
|
|
// Get SpanContext if this request is sampled
|
|
|
|
|
let maybe_span_ctx: Option<&SpanContext> = req.extensions().get();
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
// Create a new span
|
|
|
|
|
let maybe_span = maybe_span_ctx.map(|x| x.child("foo"));
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
// Create a recorder that will record start and end times
|
|
|
|
|
let recorder = SpanRecorder::new(maybe_span);
|
|
|
|
|
}
|
|
|
|
|
```
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
[layer]: https://docs.rs/tower/0.4.8/tower/trait.Layer.html
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
[Request]: https://docs.rs/http/0.2.5/http/request/struct.Request.html
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
[extension]: https://docs.rs/http/0.2.5/http/request/struct.Request.html#method.extensions
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Trace Exporters (trace_exporters)
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
The `trace_exporters` crate contains the logic to sink traces to upstream aggregators such as [Jaeger]. In the future,
|
|
|
|
|
we may also add [OTLP] in order to allow using [OpenTelemetry Collector] to fanout to different aggregators
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
[Jaeger]: https://www.jaegertracing.io
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
[OTLP]: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
[OpenTelemetry Collector]: https://github.com/open-telemetry/opentelemetry-collector
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
## Running Jaeger / tracing locally
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
To use, develop, or debug the distributed tracing functionality locally you can do the following:
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Step 1: Run Jaeger locally
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2022-07-26 09:40:42 +00:00
|
|
|
Follow instructions from https://www.jaegertracing.io/docs/1.26/getting-started/, which at the time of writing were
|
|
|
|
|
(simplified to what IOx needs):
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
```shell
|
|
|
|
|
docker run -d --name jaeger \
|
|
|
|
|
-p 6831:6831/udp \
|
|
|
|
|
-p 16686:16686 \
|
|
|
|
|
jaegertracing/all-in-one:1.26
|
2020-12-10 11:05:36 +00:00
|
|
|
```
|
|
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Step 2: Run IOx configured to send traces to the local Jaeger instance
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
Build IOx and run with the following environment variable set:
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2022-07-26 09:40:42 +00:00
|
|
|
```text
|
2021-09-30 08:46:23 +00:00
|
|
|
TRACES_EXPORTER=jaeger
|
|
|
|
|
TRACES_EXPORTER_JAEGER_AGENT_HOST=localhost
|
|
|
|
|
TRACES_EXPORTER_JAEGER_AGENT_PORT=6831
|
|
|
|
|
```
|
2020-12-14 10:56:45 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
For example, a command such as this should do the trick:
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
```shell
|
2022-06-23 09:24:38 +00:00
|
|
|
TRACES_EXPORTER=jaeger TRACES_EXPORTER_JAEGER_AGENT_HOST=localhost TRACES_EXPORTER_JAEGER_AGENT_PORT=6831 cargo run -- run all-in-one -v
|
2020-12-10 11:05:36 +00:00
|
|
|
```
|
|
|
|
|
|
2022-03-07 13:49:23 +00:00
|
|
|
Additional trace granularity, in particular traces with spans for each DataFusion partition, can be enabled with
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
INFLUXDB_IOX_PER_PARTITION_TRACING=1
|
|
|
|
|
```
|
|
|
|
|
|
2022-07-26 09:40:42 +00:00
|
|
|
_Some tracing setups may struggle with the size of the generated traces with this setting enabled._
|
2022-03-07 13:49:23 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Step 3: Send a request with trace context
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
For IOx to emit traces, the request must have a span context set. You can use the `--header` flag on the IOx CLI to do
|
|
|
|
|
so. For example
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
```shell
|
2022-07-26 09:40:42 +00:00
|
|
|
$ # load data
|
|
|
|
|
$ curl -v "http://127.0.0.1:8080/api/v2/write?org=26f7e5a4b7be365b&bucket=917b97a92e883afc" --data-binary @tests/fixtures/lineproto/metrics.lp
|
|
|
|
|
|
|
|
|
|
$ # run a query and start a new trace
|
|
|
|
|
$ cargo run -- query 26f7e5a4b7be365b_917b97a92e883afc 'show tables' --gen-trace-id
|
2020-12-10 11:05:36 +00:00
|
|
|
```
|
|
|
|
|
|
2021-09-30 08:46:23 +00:00
|
|
|
### Step 4: Explore Spans in the UI
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2022-07-26 09:40:42 +00:00
|
|
|
Navigate to the UI in your browser [localhost:16686/search](http://localhost:16686/search) and then chose the "iox-conductor" service from the
|
|
|
|
|
drop down:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You can then inspect the individual traces:
|
2020-12-10 11:05:36 +00:00
|
|
|
|
2022-07-26 09:40:42 +00:00
|
|
|
|
