Influxdata
/
influxdb
mirror of https://github.com/influxdata/influxdb.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

chore: lint markdown docs and use relative code references in docs (#3420) 

* chore: lint markdown docs

docs: relative source file references in markdown docs

* chore: review feedback
pull/24376/head
Raphael Taylor-Davies 2022-01-04 09:50:13 +00:00 committed by GitHub
parent f9174c483b
commit 5b71306423
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 192 additions and 143 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
.circleci/config.yml
View File

@ -335,6 +335,16 @@ jobs:
# other stuff that may have been added to main since last merge)
MERGE_BASE=$(git merge-base origin/main $CIRCLE_BRANCH) sh -c 'buf breaking --against ".git#ref=$MERGE_BASE"'
# Lint docs
docs-lint:
docker:
- image: python:3-slim-bullseye
steps:
- checkout
- run:
name: Lint docs
command: ./scripts/lint_docs.py ./docs
# Compile a cargo "release" profile binary for branches that end in `/perf`
#
# Uses the latest ci_image (influxdb/rust below) to build a release binary and
@ -458,6 +468,7 @@ workflows:
- lint
- cargo_audit
- protobuf-lint
- docs-lint
- test
- test_heappy
- test_perf
@ -473,6 +484,7 @@ workflows:
- lint
- cargo_audit
- protobuf-lint
- docs-lint
- test
- test_heappy
- test_perf

4
docs/catalog_persistence.md
View File

@ -144,7 +144,7 @@ Schemas during write are only enforced on “best effort” basis by the Mutable
### 2.4 UTF-8 Passthrough
The solution presented here will pass UTF-8 strings (for table and column names) as is. No [unicode normalization](http://www.unicode.org/reports/tr15/) or case-handling will be implemented.
The solution presented here will pass UTF-8 strings (for table and column names) as is. No [unicode normalization](https://www.unicode.org/reports/tr15/) or case-handling will be implemented.
### 2.5 Simple Transactions
@ -179,7 +179,7 @@ A transaction is addressed using the following schema:
This ensures the following properties:
- **Linear Enumeration:** Transactions can be traversed in a linear fashion
- **Uniqueness:** Transactions both complete and incomplete (e.g. due to a crash) are always unique. This allows the detection of uncomplete transactions later on.
- **Uniqueness:** Transactions both complete and incomplete (e.g. due to a crash) are always unique. This allows the detection of uncomplete transactions later on.
### 3.1 Catalog Checkpoint _· rejected_

100
docs/catalogs.md
View File

@ -9,10 +9,10 @@ Figure 1 below shows the organization of both IOx Catalog and Data. The bottom l
[^cat]: This design might be changed in the future to meet other use cases.
Since the catalog is the central information shared to all users of the database, it must be kept consistent. Each object of the catalog must be locked in either shared or exclusive mode when accessed or modified. Each catalog object also contains information to operate the chunk lifecycle, run queries, as well as measure the health of the system.
Since the catalog is the central information shared to all users of the database, it must be kept consistent. Each object of the catalog must be locked in either shared or exclusive mode when accessed or modified. Each catalog object also contains information to operate the chunk lifecycle, run queries, as well as measure the health of the system.
**Examples of information for operating the chunk lifecycle**
* A chunk includes `LifecycleAction` that keeps track of on-going action of a chunk (e.g. `compacting`, `persisting`) to avoid running the same job on the chunk.
* A chunk includes `LifecycleAction` that keeps track of on-going action of a chunk (e.g. `compacting`, `persisting`) to avoid running the same job on the chunk.
* A partition contains `PersistenceWindows` that keep track of ingested data within a partition to determine when it can be persisted. This allows IOx to receive out of order writes in their timestamps while persisting mostly in non-time overlapping Object Store files.
**Examples of information for measuring the health of the system**
@ -23,52 +23,52 @@ Since the catalog is the central information shared to all users of the database
```text
┌───────────┐
│IOx Server │ IOx Server
└───────────┘
│
┌──────────────┼───────────────┐
▼ ▼ ▼
┌───────────┐ ┌────────────┐
│Database 1 │ ... │ Database p │ Databases
└───────────┘ └────────────┘
│ │
▼ ▼
┌───────────┐
│ Catalog 1 │ ... Catalogs
└───────────┘
│
┌───────────────┼───────────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐
│ Table 1 │ ... │ Table n │ Tables
└──────────┘ └──────────┘
│ │
┌──────────────────┼────────────────────┐ │
▼ ▼ ▼ ▼
┌────────────┐ ┌────────────┐
│Partition 1 │ ... │Partition m │ ... Partitions
│(2021-12-10)│ │(2021-12-20)│
└────────────┘ └────────────┘
│ │
┌─────────▼───────┐ ┌───────────────┼─────────────────────┐
▼ ▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐┌──────────────┐ ┌──────────────┐
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 ││ Chunk 4 │ │ Chunk 5 │ Chunks
│ (Open) │ │ (Frozen) │ │ (Open) ││ (Frozen) │ │ (Persisted) │
└──────────────┘ └──────────────┘ └──────────────┘└──────────────┘ └───────┬──────┘
│ │ │ │ ┌──────▼──────┐
▼ ▼ ▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌───────────┐ ┌────────────────┐┌───────────┐┌───────────┐
│ O-MUB 1 │ │ RUB 2 │ │ O-MUB 3 │ │ F-MUB 4 ││ RUB 5 ││ OS 5 │
│ │ │ │ │ │ │ ││ ││ │ Data Chunks
│col1 col2 col3│ │ col1 col4 │ │ col1 col2 │ │ col1 col2 col3 ││ col2 col3 ││ col2 col3 │
│---- ---- ----│ │ ---- ---- │ │ ---- ---- │ │ ---- ---- ---- ││ ---- ---- ││ ---- ---- │
│---- ---- ----│ │ ---- ---- │ │ ---- ---- │ │ ---- ---- ---- ││ ---- ---- ││ ---- ---- │
└──────────────┘ │ ---- ---- │ └───────────┘ │ ---- ---- ---- ││ ---- ---- ││ ---- ---- │
└──────────────┘ │ ---- ---- ---- │└───────────┘└───────────┘
└────────────────┘
┌───────────┐
│IOx Server │ IOx Server
└───────────┘
│
┌──────────────┼───────────────┐
▼ ▼ ▼
┌───────────┐ ┌────────────┐
│Database 1 │ ... │ Database p │ Databases
└───────────┘ └────────────┘
│ │
▼ ▼
┌───────────┐
│ Catalog 1 │ ... Catalogs
└───────────┘
│
┌───────────────┼───────────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐
│ Table 1 │ ... │ Table n │ Tables
└──────────┘ └──────────┘
│ │
┌──────────────────┼────────────────────┐ │
▼ ▼ ▼ ▼
┌────────────┐ ┌────────────┐
│Partition 1 │ ... │Partition m │ ... Partitions
│(2021-12-10)│ │(2021-12-20)│
└────────────┘ └────────────┘
│ │
┌─────────▼───────┐ ┌───────────────┼─────────────────────┐
▼ ▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐┌──────────────┐ ┌──────────────┐
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 ││ Chunk 4 │ │ Chunk 5 │ Chunks
│ (Open) │ │ (Frozen) │ │ (Open) ││ (Frozen) │ │ (Persisted) │
└──────────────┘ └──────────────┘ └──────────────┘└──────────────┘ └───────┬──────┘
│ │ │ │ ┌──────▼──────┐
▼ ▼ ▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌───────────┐ ┌────────────────┐┌───────────┐┌───────────┐
│ O-MUB 1 │ │ RUB 2 │ │ O-MUB 3 │ │ F-MUB 4 ││ RUB 5 ││ OS 5 │
│ │ │ │ │ │ │ ││ ││ │ Data Chunks
│col1 col2 col3│ │ col1 col4 │ │ col1 col2 │ │ col1 col2 col3 ││ col2 col3 ││ col2 col3 │
│---- ---- ----│ │ ---- ---- │ │ ---- ---- │ │ ---- ---- ---- ││ ---- ---- ││ ---- ---- │
│---- ---- ----│ │ ---- ---- │ │ ---- ---- │ │ ---- ---- ---- ││ ---- ---- ││ ---- ---- │
└──────────────┘ │ ---- ---- │ └───────────┘ │ ---- ---- ---- ││ ---- ---- ││ ---- ---- │
└──────────────┘ │ ---- ---- ---- │└───────────┘└───────────┘
└────────────────┘
Figure 1: Catalog and Data Organization
```
@ -76,11 +76,11 @@ Figure 1: Catalog and Data Organization
## Rebuild a Database from its Catalog
Since the `Catalog` is the core of the database, losing the catalog means losing the database; or rebuilding a database means rebuilding its catalog. Thus to rebuild a catalog, its information needs to be saved in a durable storage.
Basically, if an IOx server goes down unexpectedly, we will lose the in-memory catalog shown in Figure 1 and all of its in-memory data chunks `O-MUBs`, `F-MUBs`, and `RUBs`. Only data of `OS` chunks are not lost. As pointed out in [IOx Data Organization and LifeCycle](data_organization_lifecycle.md) that chunk data is persisted contiguously with their loading time, `OS` chunks only include data ingested before the one in `MUBs` and `RUBs`. Thus, in principal, rebuilding the catalog includes two major steps: first rebuild the catalog that links to the already persisted `OS` chunks, then rerun ingesting data of `O-MUBs`, `F-MUBs` and `RUBs`. To perform those 2 steps, IOx saves `catalog transactions` that include minimal information of the latest stage of all valid `OS` chunks (e.g, chunks are not deleted), and `checkpoints` of when in the past to re-ingest data. At the beginning of the `Catalog rebuild`, those transactions will be read from the Object Store, and their information is used to rebuild the Catalog and rerun the necessary ingestion. Due to IOx `DataLifeCyclePolicy` that is responsible for when to trigger compacting chunk data, the rebuilt catalog may look different from the one before but as long as all the data in previous `O-MUBs`, `F-MUBs`, and `RUBs` are reloaded/recovered and tracked in the catalog, it does not matter which chunk types the data belong to. Refer to [Catalog Persistence](catalog_persistence.md) for the original design of Catalog Persistence and [CheckPoint](https://github.com/influxdata/influxdb_iox/blob/b39e01f7ba4f5d19f92862c5e87b90a40879a6c9/persistence_windows/src/checkpoint.rs) for the detailed implementation of IOx Checkpoints.
Basically, if an IOx server goes down unexpectedly, we will lose the in-memory catalog shown in Figure 1 and all of its in-memory data chunks `O-MUBs`, `F-MUBs`, and `RUBs`. Only data of `OS` chunks are not lost. As pointed out in [IOx Data Organization and LifeCycle](data_organization_lifecycle.md) that chunk data is persisted contiguously with their loading time, `OS` chunks only include data ingested before the one in `MUBs` and `RUBs`. Thus, in principal, rebuilding the catalog includes two major steps: first rebuild the catalog that links to the already persisted `OS` chunks, then rerun ingesting data of `O-MUBs`, `F-MUBs` and `RUBs`. To perform those 2 steps, IOx saves `catalog transactions` that include minimal information of the latest stage of all valid `OS` chunks (e.g, chunks are not deleted), and `checkpoints` of when in the past to re-ingest data. At the beginning of the `Catalog rebuild`, those transactions will be read from the Object Store, and their information is used to rebuild the Catalog and rerun the necessary ingestion. Due to IOx `DataLifeCyclePolicy` that is responsible for when to trigger compacting chunk data, the rebuilt catalog may look different from the one before but as long as all the data in previous `O-MUBs`, `F-MUBs`, and `RUBs` are reloaded/recovered and tracked in the catalog, it does not matter which chunk types the data belong to. Refer to [Catalog Persistence](catalog_persistence.md) for the original design of Catalog Persistence and [CheckPoint](../persistence_windows/src/checkpoint.rs) for the detailed implementation of IOx Checkpoints.
## Answer queries through the Catalog
Catalog is essential to not only lead IOx to the right chunk data for answering user queries but also critical to help read the minimal possible data. Besides the catalog structure shown in Figure 1, chunk statistics (e.g. chunk column's min, max, null count, row count) are calculated while building up the catalog and included in corresponding catalog objects. This information enables IOx to answer some queries right after reading its catalog without scanning physical chunk data as in some examples above [^query]. In addition, IOx provides system tables to let users query their catalog data such as number of tables, number of partitions per database or table, number of chunks per database or tables or partition, number of each chunk type and so on.
[^query]: Chunk pruning, query planning, optimization, and execution are beyond the scope of this document.
Now we know [IOx Data Organization and LifeCycle](data_organization_lifecycle.md) and Catalog, let us move to next topics: **IOx Transactions and Locks** (to be written and linked)
Now we know [IOx Data Organization and LifeCycle](data_organization_lifecycle.md) and Catalog, let us move to next topics: **IOx Transactions and Locks** (to be written and linked)

180
docs/data_organization_lifecycle.md
View File

@ -3,43 +3,43 @@
## Data Organization
Figure 1 illustrates an `IOx Server` which is a columnar database management system (DBMS). An IOx Server includes many `databases`, each represents an isolated dataset from an organization or user. The IOx Server in Figure 1 consists of `p` databases. Each database has as many `tables` as needed. Data of each table is partitioned on a specified partition key which is an expression of the table column(s). In the example of Figure 1, `Table 1` is partitioned by date which is an expression on a time column of `Table 1`. `Partition` data is physically split into many chunks depending on the table's flow of ingested data which will be described in the next section, Data Life Cycle. Each chunk contains a subset of rows of its table partition on a subset of columns of the table. For example, `Chunk 1` has 2 rows of data on columns `col1`, `col2`, and `col3` while `Chunk 2` includes 3 rows on `col1` and `col4`. Since every chunk can consist of data of the same or different columns, a chunk has it own `schema` defined with it. `Chunk 1`'s schema is {`col1`, `col2`, `col3`} (and their corresponding data types) and `Chunk 2`'s schema is {`col1`, `col4`}. Columns with the same name, such as `col1`, represent the same column and must have the same data type across all chunks within the table.
Figure 1 illustrates an `IOx Server` which is a columnar database management system (DBMS). An IOx Server includes many `databases`, each represents an isolated dataset from an organization or user. The IOx Server in Figure 1 consists of `p` databases. Each database has as many `tables` as needed. Data of each table is partitioned on a specified partition key which is an expression of the table column(s). In the example of Figure 1, `Table 1` is partitioned by date which is an expression on a time column of `Table 1`. `Partition` data is physically split into many chunks depending on the table's flow of ingested data which will be described in the next section, Data Life Cycle. Each chunk contains a subset of rows of its table partition on a subset of columns of the table. For example, `Chunk 1` has 2 rows of data on columns `col1`, `col2`, and `col3` while `Chunk 2` includes 3 rows on `col1` and `col4`. Since every chunk can consist of data of the same or different columns, a chunk has it own `schema` defined with it. `Chunk 1`'s schema is {`col1`, `col2`, `col3`} (and their corresponding data types) and `Chunk 2`'s schema is {`col1`, `col4`}. Columns with the same name, such as `col1`, represent the same column and must have the same data type across all chunks within the table.
```text
┌───────────┐
│IOx Server │ IOx Server
└───────────┘
│
┌──────────────┼────────────────┐
▼ ▼ ▼
┌───────────┐ ┌────────────┐
│Database 1 │ ... │ Database p │ Databases
└───────────┘ └────────────┘
│
┌──────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐
│ Table 1 │ ... │ Table n │ Tables
└──────────┘ └──────────┘
│ │
┌──────────────┼──────────────┐ │
▼ ▼ ▼ ▼
┌────────────┐ ┌────────────┐
│Partition 1 │ ... │Partition m │ ... Partitions
│(2021-12-10)│ │(2021-12-20)│
└────────────┘ └──────┬─────┘
│ │
┌─────────────┼─────────────┐ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌───────────┐
│ Chunk 1 │ ... │ Chunk 2 │ ... Chunks
│ │ │ │
│col1 col2 col3│ │ col1 col4 │
│---- ---- ----│ │ ---- ---- │
│---- ---- ----│ │ ---- ---- │
└──────────────┘ │ ---- ---- │
└───────────┘
```text
┌───────────┐
│IOx Server │ IOx Server
└───────────┘
│
┌──────────────┼────────────────┐
▼ ▼ ▼
┌───────────┐ ┌────────────┐
│Database 1 │ ... │ Database p │ Databases
└───────────┘ └────────────┘
│
┌──────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐
│ Table 1 │ ... │ Table n │ Tables
└──────────┘ └──────────┘
│ │
┌──────────────┼──────────────┐ │
▼ ▼ ▼ ▼
┌────────────┐ ┌────────────┐
│Partition 1 │ ... │Partition m │ ... Partitions
│(2021-12-10)│ │(2021-12-20)│
└────────────┘ └──────┬─────┘
│ │
┌─────────────┼─────────────┐ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌───────────┐
│ Chunk 1 │ ... │ Chunk 2 │ ... Chunks
│ │ │ │
│col1 col2 col3│ │ col1 col4 │
│---- ---- ----│ │ ---- ---- │
│---- ---- ----│ │ ---- ---- │
└──────────────┘ │ ---- ---- │
└───────────┘
Figure 1: Data organization in an IOx Server
```
@ -48,7 +48,7 @@ Chunk is considered the smallest block of data in IOx and the central discussion
[^dup]: The detail of `duplication` and `deduplication` during compaction and query are parts of a large topic that deserve another document.
### Chunk Types
A `Chunk` in IOx is an abstract object defined in the code as a [DbChunk](https://github.com/influxdata/influxdb_iox/blob/12c40b0f0f93e94e483015f9104639a1f766d594/server/src/db/chunk.rs#L78). To optimize the Data LifeCycle and Query Performance, IOx implements these types of physical chunks for a DbChunk: O-MUB, F-MUB, RUB, OS.
A `Chunk` in IOx is an abstract object defined in the code as a [DbChunk](../db/src/chunk.rs). To optimize the Data LifeCycle and Query Performance, IOx implements these types of physical chunks for a DbChunk: O-MUB, F-MUB, RUB, OS.
1. O-MUB: **O**pen **MU**table **B**uffer chunk is optimized for writes and the only chunk type that accepts ingesting data. O-MUB is an in-memory chunk but its data is not sorted and not heavily compressed.[^type]
1. F-MUB: **F**rozen **MU**table **B**uffer chunk has the same format as O-MUB (in memory, not sorted, not encoded) but it no longer accepts writes. It is used as a transition chunk while its data is being moved from optimized-for-writes to optimized-for-reads.
@ -61,7 +61,7 @@ Depending on which stage of the lifecycle a chunk is in, it will be represented
### Stages of a Chunk
Before digging into Data Lifecycle, let us look into the stages of a chunk implemented as [ChunkStage](https://github.com/influxdata/influxdb_iox/blob/76befe94ad14cd121d6fc5c58aa112997d9e211a/server/src/db/catalog/chunk.rs#L130). A chunk goes through three stages demonstrated in Figure 2: `Open`, `Frozen`, and `Persisted`.
Before digging into Data Lifecycle, let us look into the stages of a chunk implemented as [ChunkStage](../db/src/catalog/chunk.rs). A chunk goes through three stages demonstrated in Figure 2: `Open`, `Frozen`, and `Persisted`.
* When data is ingested into IOx, it will be written into an open chunk which is an `O-MUB`.
* When triggered by some manual or automatic action of the lifecycle (described in next section), the open chunk will be frozen, first to `F-MUB` then transitioned to `RUB`.
* When the `RUB` is persisted to an `OS` chunk, its stage will be moved to persisted. Unlike the `Open` and `Frozen` stages that are represented by only one type of chunk at a moment in time, the `Persisted` stage can be represented by two chunk types at a time: `RUB` and `OS` that store the same data for the purpose of query performance. When a query needs to read data of a persisted chunk stage, it will first look for `RUB`, but, if not available, will look for `OS`. `RUB` will be unloaded from the persisted stage if IOx memory runs low, and reloaded if data of that chunk is queried a lot and IOx memory is underused.
@ -112,60 +112,60 @@ Figure 3 illustrates an example of IOx Data LifeCycle with 8 actions happening a
* **AT T8**: a `Compact OS` is issued on `OS 1` and `OS 2` to produce `OS 3`.
```text
Time
───────────────────────── T1 ────────── T3 ──────────────── T3 ───── T4 ──────────────── T5 ─────────────── T6 ──────────────────────── T7 ───────────── T8 ────────▶
Ingest ┌───────┐ T1: ┌───────┐ Continue ┌─────┐
────────────▶│O-MUB 1│─────────▶│F-MUB 1│─────────▶│RUB 1│──▶│
before T1 └───────┘ Compact └───────┘compacting└─────┘ │
│
│
│
│
Ingest ┌───────┐ T2: ┌───────┐ │ T3: ┌─────┐
────────────────▶│O-MUB 2│────────▶│F-MUB 2│──┼──────────▶│RUB 2│──────────────────────────────────▶│
between T1 and T2 └───────┘ Delete └───────┘ │ Compact └─────┘ │
│ ▲ │
│ │ │
│ │ │
Ingest ┌───────┐ │ ┌───────┐ │
──────────────────▶│O-MUB 3│──┴─▶│F-MUB 3│ │
between T2 and T3 └───────┘ └───────┘ │
│
│
│ ┌─────┐
Ingest ┌───────┐ T4: ┌───────┐ │ ┌──▶│OS 1 │─────────────────────────▶│
──────────────────▶│O-MUB 4│────────▶│F-MUB 4│───▶│ │ │ └─────┘ │
between T3 and T4 └───────┘ Delete └───────┘ │ T5: ┌─────┐ │ T6: │ │
Time
───────────────────────── T1 ────────── T3 ──────────────── T3 ───── T4 ──────────────── T5 ─────────────── T6 ──────────────────────── T7 ───────────── T8 ────────▶
Ingest ┌───────┐ T1: ┌───────┐ Continue ┌─────┐
────────────▶│O-MUB 1│─────────▶│F-MUB 1│─────────▶│RUB 1│──▶│
before T1 └───────┘ Compact └───────┘compacting└─────┘ │
│
│
│
│
Ingest ┌───────┐ T2: ┌───────┐ │ T3: ┌─────┐
────────────────▶│O-MUB 2│────────▶│F-MUB 2│──┼──────────▶│RUB 2│──────────────────────────────────▶│
between T1 and T2 └───────┘ Delete └───────┘ │ Compact └─────┘ │
│ ▲ │
│ │ │
│ │ │
Ingest ┌───────┐ │ ┌───────┐ │
──────────────────▶│O-MUB 3│──┴─▶│F-MUB 3│ │
between T2 and T3 └───────┘ └───────┘ │
│
│
│ ┌─────┐
Ingest ┌───────┐ T4: ┌───────┐ │ ┌──▶│OS 1 │─────────────────────────▶│
──────────────────▶│O-MUB 4│────────▶│F-MUB 4│───▶│ │ │ └─────┘ │
between T3 and T4 └───────┘ Delete └───────┘ │ T5: ┌─────┐ │ T6: │ │
├─────────▶│RUB 3│──┼────────────▶│ │ T8: ┌─────┐
│ Compact └─────┘ │ Persist │ ├──────────▶│OS 3 │
│ ▲ │ ▲ │ ┌─────┐ │Compact OS └─────┘
│ │ │ │ └──▶│RUB 4│───▶ │
Ingest ┌───────┐ │┌───────┐ │ │ └─────┘ │ ┌─────┐ │
──────────────────▶│O-MUB 5│────▶│F-MUB 5│ │ │ │ ┌─▶│OS 2 ├─▶
between T4 and T5 └───────┘ └───────┘ │ │ │ │ └─────┘
│ │ │ T7: │
│ │ ├──────────▶│
│ │ │ Persist │
Ingest ┌───────┐ │ ┌───────┐ │ ▲ │ ┌─────┐
──────────────────▶│O-MUB 6│──────────┴─▶│F-MUB 6│ │ │ └─▶│RUB 5│
between T5 and T6 └───────┘ └───────┘ │ │ └─────┘
│ │
│ │
Ingest ┌───────┐ │ ┌───────┐
──────────────────▶│O-MUB 7│────────────────┴▶│F-MUB 7│
between T6 and T7 └───────┘ └───────┘
Ingest ┌───────┐
──────────────────▶│O-MUB 8│
after T7 └───────┘
Figure 3: Example of
│ │ │ │ └──▶│RUB 4│───▶ │
Ingest ┌───────┐ │┌───────┐ │ │ └─────┘ │ ┌─────┐ │
──────────────────▶│O-MUB 5│────▶│F-MUB 5│ │ │ │ ┌─▶│OS 2 ├─▶
between T4 and T5 └───────┘ └───────┘ │ │ │ │ └─────┘
│ │ │ T7: │
│ │ ├──────────▶│
│ │ │ Persist │
Ingest ┌───────┐ │ ┌───────┐ │ ▲ │ ┌─────┐
──────────────────▶│O-MUB 6│──────────┴─▶│F-MUB 6│ │ │ └─▶│RUB 5│
between T5 and T6 └───────┘ └───────┘ │ │ └─────┘
│ │
│ │
Ingest ┌───────┐ │ ┌───────┐
──────────────────▶│O-MUB 7│────────────────┴▶│F-MUB 7│
between T6 and T7 └───────┘ └───────┘
Ingest ┌───────┐
──────────────────▶│O-MUB 8│
after T7 └───────┘
Figure 3: Example of
```
Now we understand chunks and their life cycle, let us move to next topic: [IOx Catalog: The Metadata for Operating a Database](catalogs.md)
Now we understand chunks and their life cycle, let us move to next topic: [IOx Catalog: The Metadata for Operating a Database](catalogs.md)

2
docs/logging.md
View File

@ -35,7 +35,7 @@ $ ./influxdb_iox run database --log-filter debug --log-format logfmt
IOx makes use of Rust's [tracing](https://docs.rs/tracing) ecosystem to output application logs to stdout. It
additionally makes use of [tracing-log](https://docs.rs/tracing-log) to ensure that crates writing events to
the [log](docs.rs/log/) facade are also rendered to stdout.
the [log](https://docs.rs/log/) facade are also rendered to stdout.
### Macros

37
scripts/lint_docs.py Executable file
View File

@ -0,0 +1,37 @@
#!/usr/bin/env python3
import argparse
import re
import sys
from pathlib import Path
parser = argparse.ArgumentParser(description="Lint docs")
parser.add_argument("docs", help="docs directory")
args = parser.parse_args()
regex = re.compile('\[.+\]\((?P<link>[^)]+)\)')
success = True
for md_path in sorted(Path(args.docs).rglob("*.md")):
print(f"Checking \"{md_path}\"")
with open(md_path) as file:
links = {match.group('link'): idx + 1 for (idx, line) in enumerate(file) for match in regex.finditer(line)}
for link, line_number in links.items():
if link.startswith('https://') or link.startswith('#'):
continue
if link.startswith('http://'):
print(f"FAIL: Non-SSL URL {link} in {md_path}:{line_number}", file=sys.stderr)
success = False
continue
if not Path(md_path.parent, link).exists():
print(f"FAIL: Link {link} not found for {md_path}:{line_number}", file=sys.stderr)
success = False
if not success:
exit(1)
print("OK")