New InfluxDB admin section (#4584)
* work with InfluxDB logs * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: noramullen1 <42354779+noramullen1@users.noreply.github.com> * Manage internals – TSI index (#4570) * WIP manage internals * WIP manage tsi * add TSI management docs, related to #827 * Apply suggestions from code review Co-authored-by: kelseiv <47797004+kelseiv@users.noreply.github.com> Co-authored-by: kelseiv <47797004+kelseiv@users.noreply.github.com> Co-authored-by: kelseiv <47797004+kelseiv@users.noreply.github.com> Co-authored-by: noramullen1 <42354779+noramullen1@users.noreply.github.com>pull/4587/head
Scott Anderson
GitHub
parent
528ecd1bc9
commit
3c77f448ae
|
|
@ -47,6 +47,11 @@ a.btn {
|
|||
margin-right: .5rem;
|
||||
font-size: 1.1rem;
|
||||
}
|
||||
|
||||
&.small-plus {
|
||||
padding: .25em;
|
||||
line-height: .65rem;
|
||||
}
|
||||
}
|
||||
|
||||
///////////////////////////// InfluxDB URL Triggers ////////////////////////////
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ description: >
|
|||
Switch from one InfluxDB Cloud account to another and set a default account.
|
||||
menu:
|
||||
influxdb_cloud:
|
||||
name: Switch InfluxDB accounts
|
||||
name: Switch InfluxDB organizations
|
||||
parent: Account management
|
||||
weight: 105
|
||||
---
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ menu:
|
|||
influxdb_1_7:
|
||||
weight: 60
|
||||
parent: Administration
|
||||
v2: /influxdb/v2.4/admin/internals/tsi/rebuild-index/
|
||||
---
|
||||
|
||||
The InfluxDB [Time Series Index (TSI)](/influxdb/v1.7/concepts/tsi-details/)
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ menu:
|
|||
influxdb_1_8:
|
||||
weight: 60
|
||||
parent: Administration
|
||||
v2: /influxdb/v2.4/admin/internals/tsi/rebuild-index/
|
||||
---
|
||||
|
||||
The InfluxDB [Time Series Index (TSI)](/influxdb/v1.8/concepts/tsi-details/)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,13 @@
|
|||
---
|
||||
title: Administer InfluxDB
|
||||
description: >
|
||||
Use the InfluxDB API, user interface (UI), and CLIs to perform administrative
|
||||
tasks in InfluxDB.
|
||||
menu: influxdb_2_4
|
||||
weight: 18
|
||||
---
|
||||
|
||||
Use the InfluxDB API, user interface (UI), and CLIs to perform administrative
|
||||
tasks in InfluxDB.
|
||||
|
||||
{{< children >}}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
---
|
||||
title: Manage InfluxDB internal systems
|
||||
description: >
|
||||
Manage the internal systems of InfluxDB such as the Time Series Index (TSI),
|
||||
the time-structured merge tree (TSM) storage engine, and the write-ahead log (WAL).
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
name: Manage internal systems
|
||||
parent: Administer InfluxDB
|
||||
weight: 20
|
||||
cascade:
|
||||
influxdb/v2.4/tags: [storage, internals]
|
||||
---
|
||||
|
||||
Manage InfluxDB internal systems, including the time series index (TSI), time-structured merge tree (TSM) storage engine, and write-ahead log (WAL).
|
||||
|
||||
{{< children >}}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
---
|
||||
title: Manage the InfluxDB time series index (TSI)
|
||||
description: >
|
||||
The InfluxDB [time series index (TSI)](/influxdb/v2.4/reference/internals/storage-engine/#time-series-index-tsi)
|
||||
indexes or caches measurement and tag data to ensure queries are performant.
|
||||
Use the `influxd inspect` command to manage the TSI index.
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
name: Manage TSI indexes
|
||||
parent: Manage internal systems
|
||||
weight: 101
|
||||
---
|
||||
|
||||
The InfluxDB [time series index (TSI)](/influxdb/v2.4/reference/internals/storage-engine/#time-series-index-tsi)
|
||||
indexes or caches measurement and tag data to ensure queries are performant.
|
||||
|
||||
{{< children >}}
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
---
|
||||
title: Inspect TSI indexes
|
||||
description: >
|
||||
Use the `influxd inspect` command to inspect the InfluxDB TSI index.
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
parent: Manage TSI indexes
|
||||
related:
|
||||
- /influxdb/v2.4/reference/internals/storage-engine/
|
||||
- /influxdb/v2.4/reference/internals/file-system-layout/
|
||||
- /influxdb/v2.4/reference/cli/influxd/inspect/dump-tsi/
|
||||
- /influxdb/v2.4/reference/cli/influxd/inspect/export-index/
|
||||
- /influxdb/v2.4/reference/cli/influxd/inspect/report-tsi/
|
||||
---
|
||||
|
||||
Use the `influxd inspect` command to inspect the InfluxDB [time series index (TSI)](/influxdb/v2.4/reference/internals/storage-engine/#time-series-index-tsi).
|
||||
|
||||
- [Output information about TSI index files](#output-information-about-tsi-index-files)
|
||||
- [Output raw series data stored in the index](#output-raw-series-data-stored-in-the-index)
|
||||
- [Output measurement data stored in the index](#output-measurement-data-stored-in-the-index)
|
||||
- [Export TSI index data as SQL](#export-tsi-index-data-as-sql)
|
||||
- [Report the cardinality of TSI files](#report-the-cardinality-of-tsi-files)
|
||||
|
||||
## Output information about TSI index files
|
||||
|
||||
Use the [`influxd inspect dump-tsi` command](/influxdb/v2.4/reference/cli/influxd/inspect/dump-tsi/)
|
||||
to output low-level details about TSI index (`tsi1`) files.
|
||||
|
||||
Provide the following:
|
||||
|
||||
- ({{< req >}}) `--series-file` flag with the path to bucket's
|
||||
[`_series` directory](/influxdb/v2.4/reference/internals/file-system-layout/#tsm-directories-and-files-layout).
|
||||
- ({{< req >}}) Path to the shard's
|
||||
[`index` directory](/influxdb/v2.4/reference/internals/file-system-layout/#tsm-directories-and-files-layout)
|
||||
|
||||
```sh
|
||||
influxd inspect dump-tsi \
|
||||
--series-file ~/.influxdbv2/engine/data/056d83f962a08461/_series \
|
||||
~/.influxdbv2/engine/data/056d83f962a08461/autogen/1023/index
|
||||
```
|
||||
|
||||
{{< expand-wrapper >}}
|
||||
{{% expand "View example output" %}}
|
||||
```
|
||||
[LOG FILE] L0-00000006.tsl
|
||||
Series: 0
|
||||
Measurements: 0
|
||||
Tag Keys: 0
|
||||
Tag Values: 0
|
||||
|
||||
[INDEX FILE] L3-00000008.tsi
|
||||
Measurements: 3
|
||||
Series data size: 0 (0.0b)
|
||||
Bytes per series: 0.0b
|
||||
Tag Keys: 15
|
||||
Tag Values: 1025
|
||||
Series: 1700
|
||||
Series data size: 0 (0.0b)
|
||||
Bytes per series: 0.0b
|
||||
|
||||
[LOG FILE] L0-00000010.tsl
|
||||
Series: 0
|
||||
Measurements: 0
|
||||
Tag Keys: 0
|
||||
Tag Values: 0
|
||||
|
||||
[INDEX FILE] L2-00000011.tsi
|
||||
Measurements: 1
|
||||
Series data size: 0 (0.0b)
|
||||
Bytes per series: 0.0b
|
||||
Tag Keys: 5
|
||||
Tag Values: 9
|
||||
Series: 10
|
||||
Series data size: 0 (0.0b)
|
||||
Bytes per series: 0.0b
|
||||
```
|
||||
{{% /expand %}}
|
||||
{{< /expand-wrapper >}}
|
||||
|
||||
### Output raw series data stored in the index
|
||||
|
||||
To output raw series data stored in index files, include the `--series` flag with
|
||||
the `influxd inspect dump-tsi` command:
|
||||
|
||||
```sh
|
||||
influxd inspect dump-tsi \
|
||||
--series \
|
||||
--series-file ~/.influxdbv2/engine/data/056d83f962a08461/_series \
|
||||
~/.influxdbv2/engine/data/056d83f962a08461/autogen/1023/index
|
||||
```
|
||||
|
||||
{{< expand-wrapper >}}
|
||||
{{% expand "View example output" %}}
|
||||
```
|
||||
earthquake,code=6000iuad,id=us6000iuad,magType=mww,net=us,title=M\ 5.2\ -\ 101\ km\ SE\ of\ Palca\,\ Peru
|
||||
earthquake,code=71377273,id=pr71377273,magType=md,net=pr,title=M\ 1.9\ -\ Puerto\ Rico\ region
|
||||
earthquake,code=73794611,id=nc73794611,magType=md,net=nc,title=M\ 0.6\ -\ 13km\ ESE\ of\ Mammoth\ Lakes\,\ CA
|
||||
earthquake,code=40361800,id=ci40361800,magType=ml,net=ci,title=M\ 1.3\ -\ 12km\ SE\ of\ Olancha\,\ CA
|
||||
earthquake,code=6000itfk,id=us6000itfk,magType=mb,net=us,title=M\ 4.4\ -\ Mindanao\,\ Philippines
|
||||
earthquake,code=2022ucrr,id=ok2022ucrr,magType=ml,net=ok,title=M\ 1.4\ -\ 4\ km\ SSE\ of\ Dover\,\ Oklahoma
|
||||
earthquake,code=73792706,id=nc73792706,magType=md,net=nc,title=M\ 0.6\ -\ 7km\ W\ of\ Cobb\,\ CA
|
||||
earthquake,code=6000isjn,id=us6000isjn,magType=mww,net=us,title=M\ 5.5\ -\ 69\ km\ E\ of\ Hualien\ City\,\ Taiwan
|
||||
earthquake,code=022d8mp4dd,id=ak022d8mp4dd,magType=ml,net=ak,title=M\ 1.3\ -\ Southern\ Alaska
|
||||
earthquake,code=022dbrb8vb,id=ak022dbrb8vb,magType=ml,net=ak,title=M\ 1.6\ -\ 37\ km\ NE\ of\ Paxson\,\ Alaska
|
||||
earthquake,code=6000iu2e,id=us6000iu2e,magType=mb,net=us,title=M\ 4.1\ -\ 81\ km\ WSW\ of\ San\ Antonio\ de\ los\ Cobres\,\ Argentina
|
||||
```
|
||||
{{% /expand %}}
|
||||
{{< /expand-wrapper >}}
|
||||
|
||||
### Output measurement data stored in the index
|
||||
|
||||
To output measurement information stored in index files, include the `--measurement`
|
||||
flag with the `influxd inspect dump-tsi` command:
|
||||
|
||||
```sh
|
||||
influxd inspect dump-tsi \
|
||||
--measurements \
|
||||
--series-file ~/.influxdbv2/engine/data/056d83f962a08461/_series \
|
||||
~/.influxdbv2/engine/data/056d83f962a08461/autogen/1023/index
|
||||
```
|
||||
|
||||
{{< expand-wrapper >}}
|
||||
{{% expand "View example output" %}}
|
||||
```
|
||||
Measurement
|
||||
earthquake
|
||||
explosion
|
||||
quarry blast
|
||||
|
||||
|
||||
Measurement
|
||||
earthquake
|
||||
explosion
|
||||
ice quake
|
||||
quarry blast
|
||||
|
||||
|
||||
Measurement
|
||||
earthquake
|
||||
explosion
|
||||
```
|
||||
{{% /expand %}}
|
||||
{{< /expand-wrapper >}}
|
||||
|
||||
## Export TSI index data as SQL
|
||||
|
||||
Use the [`influxd inspect export-index` command](/influxdb/v2.4/reference/cli/influxd/inspect/export-index/)
|
||||
to export an index in SQL format for easier inspection and debugging.
|
||||
Provide the following:
|
||||
|
||||
- `--series-path` flag with the path to the bucket's
|
||||
[`_series` directory](/influxdb/v2.4/reference/internals/file-system-layout/#tsm-directories-and-files-layout).
|
||||
- `--index-path` flag with the path to the shard's
|
||||
[`index` directory](/influxdb/v2.4/reference/internals/file-system-layout/#tsm-directories-and-files-layout).
|
||||
|
||||
```sh
|
||||
influxd inspect export-index \
|
||||
--series-path ~/.influxdbv2/engine/data/056d83f962a08461/_series \
|
||||
--index-path ~/.influxdbv2/engine/data/056d83f962a08461/autogen/1023/index
|
||||
```
|
||||
|
||||
{{< expand-wrapper >}}
|
||||
{{% expand "View example output" %}}
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS measurement_series (
|
||||
name TEXT NOT NULL,
|
||||
series_id INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tag_value_series (
|
||||
name TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
series_id INTEGER NOT NULL
|
||||
);
|
||||
|
||||
BEGIN TRANSACTION;
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26920);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26928);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26936);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26944);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26952);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26960);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26968);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26976);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26984);
|
||||
INSERT INTO measurement_series (name, series_id) VALUES ('earthquake', 26992);
|
||||
COMMIT;
|
||||
```
|
||||
{{% /expand %}}
|
||||
{{< /expand-wrapper >}}
|
||||
|
||||
## Report the cardinality of TSI files
|
||||
|
||||
Use the [`influxd inspect report-tsi` command](/influxdb/v2.4/reference/cli/influxd/inspect/report-tsi/)
|
||||
to output information about the cardinality of data in a bucket's index.
|
||||
Provide the following:
|
||||
|
||||
- `--bucket-id` with the ID of the bucket.
|
||||
|
||||
```sh
|
||||
influxd inspect report-tsi --bucket-id 056d83f962a08461
|
||||
```
|
||||
|
||||
{{< expand-wrapper >}}
|
||||
{{% expand "View example output" %}}
|
||||
```
|
||||
Summary
|
||||
Database Path: /Users/scottanderson/.influxdbv2/engine/data/056d83f962a08461
|
||||
Cardinality (exact): 101698
|
||||
|
||||
Measurement Cardinality (exact)
|
||||
|
||||
"earthquake" 99876
|
||||
"quarry blast" 1160
|
||||
"explosion" 589
|
||||
"ice quake" 58
|
||||
"other event" 10
|
||||
"chemical explosion" 2
|
||||
"rock burst" 1
|
||||
"sonic boom" 1
|
||||
"volcanic eruption" 1
|
||||
|
||||
|
||||
===============
|
||||
Shard ID: 452
|
||||
Path: /Users/scottanderson/.influxdbv2/engine/data/056d83f962a08461/autogen/452
|
||||
Cardinality (exact): 1644
|
||||
|
||||
Measurement Cardinality (exact)
|
||||
|
||||
"earthquake" 1607
|
||||
"quarry blast" 29
|
||||
"explosion" 7
|
||||
"sonic boom" 1
|
||||
===============
|
||||
|
||||
===============
|
||||
Shard ID: 453
|
||||
Path: /Users/scottanderson/.influxdbv2/engine/data/056d83f962a08461/autogen/453
|
||||
Cardinality (exact): 2329
|
||||
|
||||
Measurement Cardinality (exact)
|
||||
|
||||
"earthquake" 2298
|
||||
"quarry blast" 24
|
||||
"explosion" 7
|
||||
===============
|
||||
```
|
||||
{{% /expand %}}
|
||||
{{< /expand-wrapper >}}
|
||||
|
|
@ -0,0 +1,100 @@
|
|||
---
|
||||
title: Rebuild the TSI index
|
||||
description: >
|
||||
Flush and rebuild the TSI index to purge corrupt index files or remove indexed
|
||||
data that is out of date.
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
parent: Manage TSI indexes
|
||||
weight: 201
|
||||
related:
|
||||
- /influxdb/v2.4/reference/internals/storage-engine/
|
||||
- /influxdb/v2.4/reference/internals/file-system-layout/
|
||||
- /influxdb/v2.4/reference/cli/influxd/inspect/build-tsi/
|
||||
---
|
||||
|
||||
In some cases, it may be necessary to flush and rebuild the TSI index.
|
||||
For example, purging corrupt index files or removing outdated indexed data.
|
||||
|
||||
To rebuild your InfluxDB TSI index:
|
||||
|
||||
1. **Stop the InfluxDB (`influxd`) process**.
|
||||
|
||||
{{% warn %}}
|
||||
Rebuilding the TSI index while the `influxd` is running could prevent some data
|
||||
from being queryable.
|
||||
{{% /warn %}}
|
||||
|
||||
2. Navigate to the `data` directory in your
|
||||
[InfluxDB engine path](/influxdb/v2.4/reference/internals/file-system-layout/).
|
||||
_The engine path depends on your operating system or
|
||||
[custom engine path setting](/influxdb/v2.4/reference/config-options/#engine-path)._
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[macOS & Linux](#)
|
||||
[Windows (PowerShell)](#)
|
||||
{{% /code-tabs %}}
|
||||
{{% code-tab-content %}}
|
||||
```sh
|
||||
cd ~/.influxdbv2/engine/data/
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```powershell
|
||||
cd -Path 'C:\%USERPROFILE%\.influxdbv2\engine\data\'
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
3. **Delete all `_series` directories in your InfluxDB `data` directory.**
|
||||
By default, `_series` directories are are stored at `/data/<bucket-id>/_series`,
|
||||
but check for and remove `_series` directories throughout the
|
||||
`data` directory.
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[macOS & Linux](#)
|
||||
[Windows (PowerShell)](#)
|
||||
{{% /code-tabs %}}
|
||||
{{% code-tab-content %}}
|
||||
```sh
|
||||
find . -type d -name _series -exec -delete
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```powershell
|
||||
get-childitem -Include _series -Recurse -force | Remove-Item -Force -Recurse
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
|
||||
4. **Delete all `index` directories.** By default, `index` directories are stored at
|
||||
`/data/<bucket-id>/autogen/<shard-id>/index`, but check for and remove
|
||||
`index` directories throughout the `data` directory.
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[macOS & Linux](#)
|
||||
[Windows (PowerShell)](#)
|
||||
{{% /code-tabs %}}
|
||||
{{% code-tab-content %}}
|
||||
```sh
|
||||
find . -type d -name index -exec -delete
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```powershell
|
||||
get-childitem -Include index -Recurse -force | Remove-Item -Force -Recurse
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
|
||||
5. Use the [`influxd inspect build-tsi` command](/influxdb/v2.4/reference/cli/influxd/inspect/build-tsi/)
|
||||
to rebuild the TSI index.
|
||||
|
||||
```sh
|
||||
influxd inspect build-tsi
|
||||
```
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
---
|
||||
title: Manage InfluxDB TSM files
|
||||
description: >
|
||||
...
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
name: Manage TSM files
|
||||
parent: Manage internal systems
|
||||
weight: 101
|
||||
draft: true
|
||||
---
|
||||
|
||||
<!--
|
||||
|
||||
Marked as draft. Placeholder for future content.
|
||||
|
||||
-->
|
||||
|
||||
- influxd inspect delete-tsm Deletes a measurement from a raw tsm file.
|
||||
- influxd inspect dump-tsm Dumps low-level details about tsm1 files
|
||||
- influxd inspect export-lp Export TSM data as line protocol
|
||||
- influxd inspect report-tsm Run TSM report
|
||||
- influxd inspect verify-tombstone Verify the integrity of tombstone files
|
||||
- influxd inspect verify-tsm Verifies the integrity of TSM files
|
||||
- influxd inspect verify-wal Check for WAL corruption
|
||||
- influxd inspect verify-tombstone Verify the integrity of tombstone files
|
||||
- influxd inspect verify-seriesfile Verifies the integrity of series files.
|
||||
- influxd inspect build-tsi --compact-series-file (Compact a series file without rebuilding the index)
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
---
|
||||
title: Manage InfluxDB WAL files
|
||||
description: >
|
||||
...
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
name: Manage WAL files
|
||||
parent: Manage internal systems
|
||||
weight: 101
|
||||
draft: true
|
||||
---
|
||||
|
||||
<!--
|
||||
|
||||
Marked as draft. Placeholder for future content.
|
||||
|
||||
-->
|
||||
|
||||
dump-wal Dumps TSM data from WAL files
|
||||
verify-wal Check for WAL corruption
|
||||
|
|
@ -0,0 +1,275 @@
|
|||
---
|
||||
title: Manage InfluxDB logs
|
||||
description: >
|
||||
Learn how to configure, manage, and process your InfluxDB logs.
|
||||
menu:
|
||||
influxdb_2_4:
|
||||
name: Manage logs
|
||||
parent: Administer InfluxDB
|
||||
weight: 10
|
||||
---
|
||||
|
||||
Learn how to configure, manage, and process your InfluxDB logs:
|
||||
|
||||
- [Configure your InfluxDB log location](#configure-your-influxdb-log-location)
|
||||
- [Configure your log level](#configure-your-log-level)
|
||||
- [Enable the Flux query log](#enable-the-flux-query-log)
|
||||
- [Use external tools to manage and process logs](#use-external-tools-to-manage-and-process-logs)
|
||||
- [Log formats](#log-formats)
|
||||
|
||||
## Configure your InfluxDB log location
|
||||
|
||||
By default, InfluxDB outputs all logs to **stdout**. To view InfluxDB logs,
|
||||
view the output of the [`influxd`](/influxdb/v2.4/reference/cli/influxd/) process.
|
||||
|
||||
- [Write logs to a file](#write-logs-to-a-file)
|
||||
- [Logs when running InfluxDB as a service](#logs-when-running-influxdb-as-a-service)
|
||||
|
||||
### Write logs to a file
|
||||
|
||||
To write InfluxDB logs to a file, redirect **stdout** to a file when starting
|
||||
the InfluxDB service ([`influxd`](/influxdb/v2.4/reference/cli/influxd/)).
|
||||
|
||||
```sh
|
||||
influxd 1> /path/to/influxdb.log
|
||||
```
|
||||
|
||||
{{% note %}}
|
||||
When logging to a file, InfluxDB uses the [logfmt](#logfmt) format.
|
||||
{{% /note %}}
|
||||
|
||||
### Logs when running InfluxDB as a service
|
||||
|
||||
If you use a service manager to run InfluxDB, the service manager determines the location of logs.
|
||||
|
||||
{{< tabs-wrapper >}}
|
||||
{{% tabs %}}
|
||||
[systemd](#)
|
||||
[sysvinit](#)
|
||||
{{% /tabs %}}
|
||||
<!------------------------------- BEGIN systemd ------------------------------->
|
||||
{{% tab-content %}}
|
||||
|
||||
Most Linux systems direct logs to the `systemd` journal.
|
||||
To access these logs, use the following command:
|
||||
|
||||
```sh
|
||||
sudo journalctl -u influxdb.service
|
||||
```
|
||||
|
||||
For more information, see the [journald.conf documentation](https://www.freedesktop.org/software/systemd/man/journald.conf.html).
|
||||
|
||||
{{% /tab-content %}}
|
||||
<!-------------------------------- END systemd -------------------------------->
|
||||
<!------------------------------- BEGIN sysvinit ------------------------------>
|
||||
{{% tab-content %}}
|
||||
|
||||
When InfluxDB is run as a service, **stdout** is discarded by default (sent to `/dev/null`).
|
||||
To write logs to a file:
|
||||
|
||||
1. Open the InfluxDB startup script (`/etc/default/influxdb`) in a text editor.
|
||||
2. Set the `STDOUT` environment variable to the path where you want to store
|
||||
the InfluxDB logs. For example:
|
||||
|
||||
```conf
|
||||
STDOUT=/var/log/influxdb/influxd.log
|
||||
```
|
||||
|
||||
3. Save the changes to the startup script.
|
||||
4. Restart the InfluxDB service to apply the changes.
|
||||
|
||||
```sh
|
||||
service influxdb restart
|
||||
```
|
||||
|
||||
{{% /tab-content %}}
|
||||
<!-------------------------------- END sysvinit ------------------------------->
|
||||
{{< /tabs-wrapper >}}
|
||||
|
||||
## Configure your log level
|
||||
|
||||
Use the [`log-level` InfluxDB configuration option](/influxdb/v2.4/reference/config-options/#log-level)
|
||||
to specify the log levels the InfluxDB service outputs.
|
||||
InfluxDB supports the following log levels:
|
||||
|
||||
- **debug**: Output logs with debug, info, and error log levels.
|
||||
- **info**: _(Default)_ Output logs with info and error log levels.
|
||||
- **error**: Output logs with the error log level only.
|
||||
|
||||
{{< tabs-wrapper >}}
|
||||
{{% tabs "small" %}}
|
||||
[influxd flag](#)
|
||||
[Environment variable](#)
|
||||
[InfluxDB configuration file](#)
|
||||
{{% /tabs %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
```sh
|
||||
influxd --log-level=info
|
||||
```
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
```sh
|
||||
export INFLUXD_LOG_LEVEL=info
|
||||
```
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[YAML](#)
|
||||
[TOML](#)
|
||||
[JSON](#)
|
||||
{{% /code-tabs %}}
|
||||
{{% code-tab-content %}}
|
||||
```yml
|
||||
log-level: info
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```toml
|
||||
log-level = "info"
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```json
|
||||
{
|
||||
"log-level": "info"
|
||||
}
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{< /tabs-wrapper >}}
|
||||
|
||||
_For information about configuring InfluxDB, see [InfluxDB configuration options](/influxdb/v2.4/reference/config-options/)._
|
||||
|
||||
## Enable the Flux query log
|
||||
|
||||
Use the [`flux-log-enabled` configuration option](/influxdb/v2.4/reference/config-options/#flux-log-enabled)
|
||||
to enable Flux query logging. InfluxDB outputs Flux query logs to **stdout**
|
||||
with all other InfluxDB logs.
|
||||
|
||||
{{< tabs-wrapper >}}
|
||||
{{% tabs "small" %}}
|
||||
[influxd flag](#)
|
||||
[Environment variable](#)
|
||||
[InfluxDB configuration file](#)
|
||||
{{% /tabs %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
```sh
|
||||
influxd --flux-log-enabled
|
||||
```
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
```sh
|
||||
export INFLUXD_FLUX_LOG_ENABLED=true
|
||||
```
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{% tab-content %}}
|
||||
|
||||
{{< code-tabs-wrapper >}}
|
||||
{{% code-tabs %}}
|
||||
[YAML](#)
|
||||
[TOML](#)
|
||||
[JSON](#)
|
||||
{{% /code-tabs %}}
|
||||
{{% code-tab-content %}}
|
||||
```yml
|
||||
flux-log-enabled: true
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```toml
|
||||
flux-log-enabled = true
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{% code-tab-content %}}
|
||||
```json
|
||||
{
|
||||
"flux-log-enabled": true
|
||||
}
|
||||
```
|
||||
{{% /code-tab-content %}}
|
||||
{{< /code-tabs-wrapper >}}
|
||||
|
||||
{{% /tab-content %}}
|
||||
|
||||
{{< /tabs-wrapper >}}
|
||||
|
||||
_For information about configuring InfluxDB, see [InfluxDB configuration options](/influxdb/v2.4/reference/config-options/)._
|
||||
|
||||
|
||||
## Use external tools to manage and process logs
|
||||
|
||||
Use the following popular tools to manage and process InfluxDB logs:
|
||||
|
||||
### logrotate
|
||||
|
||||
[logrotate](https://github.com/logrotate/logrotate) simplifies the
|
||||
administration of log files and provides automatic rotation compression, removal
|
||||
and mailing of log files. Logrotate can be set to handle a log file hourly,
|
||||
daily, weekly, monthly or when the log file gets to a certain size.
|
||||
|
||||
### hutils
|
||||
|
||||
[hutils](https://blog.heroku.com/hutils-explore-your-structured-data-logs) is a
|
||||
collection of command line utilities for working with logs with [logfmt](#logfmt)
|
||||
encoding, including:
|
||||
|
||||
- **lcut**: Extracts values from a logfmt trace based on a specified field name.
|
||||
- **lfmt**: Reformats and highlights key sections of logfmt lines.
|
||||
- **ltap**: Accesses messages from log providers in a consistent way to allow
|
||||
easy parsing by other utilities that operate on logfmt traces.
|
||||
- **lviz**: Visualizes logfmt output by building a tree out of a dataset
|
||||
combining common sets of key-value pairs into shared parent nodes.
|
||||
|
||||
### lnav (Log File Navigator)
|
||||
|
||||
[lnav (Log File Navigator)](http://lnav.org/) is an advanced log file viewer useful for watching
|
||||
and analyzing log files from a terminal.
|
||||
The lnav viewer provides a single log view, automatic log format detection,
|
||||
filtering, timeline view, pretty-print view, and querying logs using SQL.
|
||||
|
||||
|
||||
## Log formats
|
||||
|
||||
InfluxDB outputs logs in one of two formats depending on the location of where
|
||||
logs are output.
|
||||
|
||||
- [Console/TTY](#consoletty)
|
||||
- [logfmt](#logfmt)
|
||||
|
||||
### Console/TTY
|
||||
|
||||
**When logging to a terminal or other TTY devices**, InfluxDB uses a console-friendly format.
|
||||
|
||||
##### Example console/TTY format
|
||||
```sh
|
||||
2022-09-29T21:58:29.936355Z info Welcome to InfluxDB {"log_id": "0dEoz3C0000", "version": "dev", "commit": "663d43d210", "build_date": "2022-09-29T21:58:29Z", "log_level": "info"}
|
||||
2022-09-29T21:58:29.977671Z info Resources opened {"log_id": "0dEoz3C0000", "service": "bolt", "path": "/Users/exampleuser/.influxdbv2/influxd.bolt"}
|
||||
2022-09-29T21:58:29.977891Z info Resources opened {"log_id": "0dEoz3C0000", "service": "sqlite", "path": "/Users/exampleuser/.influxdbv2/influxd.sqlite"}
|
||||
2022-09-29T21:58:30.059709Z info Checking InfluxDB metadata for prior version. {"log_id": "0dEoz3C0000", "bolt_path": "/Users/exampleuser/.influxdbv2/influxd.bolt"}
|
||||
```
|
||||
|
||||
### logfmt
|
||||
|
||||
**When logging to a file**, InfluxDB uses **logfmt**, a machine-readable
|
||||
structured log format that provides simpler integrations with external tools like
|
||||
[Splunk](https://www.splunk.com/), [Papertrail](https://www.papertrail.com/),
|
||||
[Elasticsearch](https://www.elastic.co/), and other third party tools.
|
||||
|
||||
##### Example logfmt format
|
||||
```sh
|
||||
ts=2022-09-29T16:54:16.021427Z lvl=info msg="Welcome to InfluxDB" log_id=0dEYZvqG000 version=dev commit=663d43d210 build_date=2022-09-29T16:54:15Z log_level=info
|
||||
ts=2022-09-29T16:54:16.062239Z lvl=info msg="Resources opened" log_id=0dEYZvqG000 service=bolt path=/Users/exampleuser/.influxdbv2/influxd.bolt
|
||||
ts=2022-09-29T16:54:16.062457Z lvl=info msg="Resources opened" log_id=0dEYZvqG000 service=sqlite path=/Users/exampleuser/.influxdbv2/influxd.sqlite
|
||||
ts=2022-09-29T16:54:16.144430Z lvl=info msg="Checking InfluxDB metadata for prior version." log_id=0dEYZvqG000 bolt_path=/Users/exampleuser/.influxdbv2/influxd.bolt
|
||||
```
|
||||
|
|
@ -19,8 +19,8 @@ influxd inspect export-index [flags]
|
|||
```
|
||||
|
||||
## Flags
|
||||
| Flag | | Description | Input type |
|
||||
|:---- |:--- |:----------- |:----------:|
|
||||
| `-h` | `--help` | Help for the `export-index` command. | |
|
||||
| |`--index-path` | Path to the index directory. Defaults to `~/.influxdbv2/engine/index`). | string |
|
||||
| |`--series-path` | Path to series file. Defaults to `~/.influxdbv2/engine/_series`). | string |
|
||||
| Flag | | Description | Input type |
|
||||
| :--- | :-------------- | :---------------------------------------------------------------------------------------------------------------------------- | :--------: |
|
||||
| `-h` | `--help` | Help for the `export-index` command. | |
|
||||
| | `--index-path` | Path to the [`index` directory](/influxdb/v2.4/reference/internals/file-system-layout/#tsm-directories-and-files-layout). | string |
|
||||
| | `--series-path` | Path to the [`_series` directory]((/influxdb/v2.4/reference/internals/file-system-layout/#tsm-directories-and-files-layout)). | string |
|
||||
|
|
|
|||
|
|
@ -460,7 +460,7 @@ Include option to show detailed logs for Flux queries, including the following l
|
|||
|
||||
###### influxd flag
|
||||
```sh
|
||||
influxd --flux-log-enabled=true
|
||||
influxd --flux-log-enabled
|
||||
```
|
||||
|
||||
###### Environment variable
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ influxdb/v2.4/tags: [storage, internals]
|
|||
related:
|
||||
- /influxdb/v2.4/reference/internals/shards/
|
||||
- /influxdb/v2.4/reference/internals/storage-engine/
|
||||
- /influxdb/v2.4/admin/internals/
|
||||
---
|
||||
|
||||
The **InfluxDB retention enforcement service** checks for and removes data with
|
||||
|
|
|
|||
|
|
@ -9,6 +9,8 @@ menu:
|
|||
name: File system layout
|
||||
parent: InfluxDB internals
|
||||
influxdb/v2.4/tags: [storage, internals]
|
||||
related:
|
||||
- /influxdb/v2.4/admin/internals/
|
||||
---
|
||||
|
||||
The InfluxDB file system layout depends on the operating system, installation method,
|
||||
|
|
@ -24,8 +26,13 @@ The InfluxDB file structure includes of the following:
|
|||
Directory path to the [storage engine](/{{< latest "influxdb" >}}/reference/internals/storage-engine/),
|
||||
where InfluxDB stores time series data, includes the following directories:
|
||||
|
||||
- **data**: stores Time-Structured Merge Tree (TSM) files
|
||||
- **wal**: stores Write Ahead Log (WAL) files.
|
||||
- **data**: Stores time-structured merge tree (TSM) files.
|
||||
For more information about the structure of the `data` directory, see
|
||||
[TSM directories and files layout](#tsm-directories-and-files-layout).
|
||||
- **replicationq**: Store the replication queue for the [InfluxDB replication service](/influxdb/v2.4/write-data/replication/).
|
||||
- **wal**: Stores write-ahead log (WAL) files.
|
||||
For more information about the structure of the `wal` directory, see
|
||||
[WAL directories and files layout](#wal-directories-and-files-layout).
|
||||
|
||||
To customize this path, use the [engine-path](/influxdb/v2.4/reference/config-options/#engine-path)
|
||||
configuration option.
|
||||
|
|
@ -248,6 +255,7 @@ so you can easily mount separate volumes for InfluxDB 1.x and 2.x data during th
|
|||
- engine/
|
||||
- data/
|
||||
- _<span style="opacity:.4">TSM directories and files</span>_
|
||||
<a class="btn small-plus" href="#tsm-directories-and-files-layout">+</a>
|
||||
- wal/
|
||||
- _<span style="opacity:.4">WAL directories and files</span>_
|
||||
- influxd.bolt
|
||||
|
|
@ -258,3 +266,37 @@ so you can easily mount separate volumes for InfluxDB 1.x and 2.x data during th
|
|||
{{% /tab-content %}}
|
||||
<!--------------------------- END KUBERNETES CONTENT -------------------------->
|
||||
{{< /tabs-wrapper >}}
|
||||
|
||||
---
|
||||
|
||||
#### TSM directories and files layout
|
||||
|
||||
TSM directories and files are stored in the `data` directory inside the [engine path](#engine-path).
|
||||
The diagram below is **relative to the [engine path](#file-system-layout)**.
|
||||
|
||||
{{% filesystem-diagram %}}
|
||||
- <span style="opacity:.4">...</span>/data/
|
||||
- 000xX00xxXx000x0/ _<span style="opacity:.4">(bucket ID)</span>_
|
||||
- _series/ _<span style="opacity:.4">(series directory)</span>_
|
||||
- 00/ _<span style="opacity:.4">(internal shard index)</span>_
|
||||
- 0000 _<span style="opacity:.4">(internal shard index file)</span>_
|
||||
- autogen
|
||||
- 0123/ _<span style="opacity:.4">(shard ID)</span>_
|
||||
- index _<span style="opacity:.4">(index directory)</span>_
|
||||
- L0-00000001.tsl _<span style="opacity:.4">(write-ahead log for the TSI index)</span>_
|
||||
- L0-00000001.tsi _<span style="opacity:.4">(series index)</span>_
|
||||
- MANIFEST _<span style="opacity:.4">(index manifest)</span>_
|
||||
{{% /filesystem-diagram %}}
|
||||
|
||||
#### WAL directories and files layout
|
||||
|
||||
WAL directories and files are stored in the `data` directory inside the [engine path](#engine-path).
|
||||
The diagram below is **relative to the [engine path](#file-system-layout)**.
|
||||
|
||||
{{% filesystem-diagram %}}
|
||||
- <span style="opacity:.4">...</span>/wal/
|
||||
- 000xX00xxXx000x0/ _<span style="opacity:.4">(bucket ID)</span>_
|
||||
- autogen/
|
||||
- 0123/ _<span style="opacity:.4">(shard ID)</span>_
|
||||
- _01234.wal _<span style="opacity:.4">(WAL file)</span>_
|
||||
{{% /filesystem-diagram %}}
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ related:
|
|||
- /influxdb/v2.4/reference/internals/storage-engine/
|
||||
- /influxdb/v2.4/organizations/buckets/
|
||||
- /influxdb/v2.4/reference/cli/influx/bucket/
|
||||
- /influxdb/v2.4/admin/internals/
|
||||
---
|
||||
|
||||
InfluxDB organizes time series data into **shards** when storing data to disk.
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ influxdb/v2.4/tags: [storage, internals]
|
|||
products: [oss]
|
||||
related:
|
||||
- /resources/videos/tsm-engine/
|
||||
- /influxdb/v2.4/admin/internals/
|
||||
---
|
||||
|
||||
The InfluxDB storage engine ensures that:
|
||||
|
|
|
|||
Loading…
Reference in New Issue