Influxdata
/
docs-v2
mirror of https://github.com/influxdata/docs-v2.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
docs-v2/content/influxdb3/clustered/install/set-up-cluster/prerequisites.md

16 KiB
Raw Permalink Blame History

title description menu weight aliases
Set up prerequisites InfluxDB Clustered requires prerequisites such as a Kubernetes cluster, object storage, a PostgreSQL-compatitble database and more. Learn how to set up and configure the necessary prerequisites.
influxdb3_clustered
name parent
Prerequisites Set up your cluster
201
/influxdb3/clustered/install/prerequisites/

InfluxDB Clustered requires the following prerequisite external dependencies:

  • kubectl command line tool
  • Kubernetes cluster
  • kubecfg kubit operator
  • Kubernetes ingress controller
  • Object storage: AWS S3 or S3-compatible storage (including Google Cloud Storage or Azure Blob Storage) to store the InfluxDB Parquet files.
  • PostgreSQL-compatible database (AWS Aurora, hosted PostgreSQL, etc.): Stores the InfluxDB Catalog.
  • Local or attached storage: Stores the Write-Ahead Log (WAL) for InfluxDB Ingesters.

The following walks through preparing these prerequisites.

Install kubectl

Kubernetes provides the kubectl command line tool for communicating with a Kubernetes cluster's control plane. kubectl is used to manage your InfluxDB cluster.

Follow instructions to install kubectl on your local machine:

{{% note %}} InfluxDB Clustered Kubernetes deployments require kubectl 1.27 or higher. {{% /note %}}

Set up your Kubernetes cluster

  1. Deploy a Kubernetes cluster. The deployment process depends on your Kubernetes environment or Kubernetes cloud provider. Refer to the Kubernetes documentation or your cloud provider's documentation for information about deploying a Kubernetes cluster.

  2. Ensure kubectl can connect to your Kubernetes cluster. Your Manage kubeconfig file defines cluster connection credentials.

  3. Create two namespaces--influxdb and kubit. Use kubectl create namespace to create the namespaces:

    kubectl create namespace influxdb && \
kubectl create namespace kubit

  4. Install an ingress controller in the cluster and a mechanism to obtain a valid TLS certificate (for example: cert-manager or provide the certificate PEM manually out of band). To use the InfluxDB-specific ingress controller, install Ingress NGINX.

  5. Ensure your Kubernetes cluster can access the InfluxDB container registry, or, if running in an air-gapped environment, a local container registry to which you can copy the InfluxDB images.

Cluster sizing recommendation

As a starting point for a production workload, InfluxData recommends the following sizing for {{% product-name %}} components:

{{< tabs-wrapper >}} {{% tabs %}} AWS Google Cloud Platform Microsoft Azure On-Prem {{% /tabs %}} {{% tab-content %}}

  • Catalog (PostgreSQL-compatible database) (x1):
  • Ingesters and Routers (x3):
    • EC2 m6i.2xlarge (8 CPU, 32 GB RAM)
    • Local storage: minimum of 2 GB (high-speed SSD)
  • Queriers (x3):
    • EC2 m6i.2xlarge (8 CPU, 32 GB RAM)
  • Compactors (x1):
    • EC2 m6i.2xlarge (8 CPU, 32 GB RAM)
  • Kubernetes Control Plane (x1):
    • EC2 t3.large (2 CPU, 8 GB RAM)

{{% /tab-content %}} {{% tab-content %}}

  • Catalog (PostgreSQL-compatible database) (x1):
  • Ingesters and Routers (x3):
    • GCE c2-standard-8 (8 CPU, 32 GB RAM)
    • Local storage: minimum of 2 GB (high-speed SSD)
  • Queriers (x3):
    • GCE c2-standard-8 (8 CPU, 32 GB RAM)
  • Compactors (x1):
    • GCE c2-standard-8 (8 CPU, 32 GB RAM)
  • Kubernetes Control Plane (x1):
    • GCE c2d-standard-2 (2 CPU, 8 GB RAM)

{{% /tab-content %}} {{% tab-content %}}

  • Catalog (PostgreSQL-compatible database) (x1):
  • Ingesters and Routers (x3):
    • Standard_D8s_v3 (8 CPU, 32 GB RAM)
    • Local storage: minimum of 2 GB (high-speed SSD)
  • Queriers (x3):
    • Standard_D8s_v3 (8 CPU, 32 GB RAM)
  • Compactors (x1):
    • Standard_D8s_v3 (8 CPU, 32 GB RAM)
  • Kubernetes Control Plane (x1):
    • Standard_B2ms (2 CPU, 8 GB RAM)

{{% /tab-content %}} {{% tab-content %}}

  • Catalog (PostgreSQL-compatible database) (x1):
    • CPU: 4-8 cores
    • RAM: 16-32 GB
  • Ingesters and Routers (x3):
    • CPU: 8 cores
    • RAM: 32 GB
    • Local storage: 2 GB (high-speed SSD)
  • Queriers (x3):
    • CPU: 8 cores
    • RAM: 32 GB
  • Compactors (x1):
    • CPU: 8 cores
    • RAM: 32 GB
  • Kubernetes Control Plane (x1):
    • CPU: 2 cores
    • RAM: 8 GB

{{% /tab-content %}} {{< /tabs-wrapper >}}

Your sizing may need to be different based on your environment, cloud provider, and workload, but this is a reasonable starting size for your initial testing.

Install the kubecfg kubit operator

The kubecfg kubit operator (maintained by InfluxData) simplifies the installation and management of the InfluxDB Clustered package. It manages the application of the jsonnet templates used to install, manage, and update an InfluxDB cluster.

{{% note %}}

The InfluxDB Clustered Helm chart includes the kubit operator

If using the InfluxDB Clustered Helm chart to deploy your InfluxDB cluster, you do not need to install the kubit operator separately. The Helm chart installs the kubit operator. {{% /note %}}

Use kubectl to install the kubecfg kubit operator v0.0.18 or later.

kubectl apply -k 'https://github.com/kubecfg/kubit//kustomize/global?ref=v0.0.19'

Set up a Kubernetes ingress controller

Kubernetes ingress routes HTTP/S requests to services within the cluster and requires deploying an ingress controller. You can provide your own ingress or you can install Nginx Ingress Controller to use the InfluxDB-defined ingress.

{{% note %}} InfluxDB Clustered components use gRPC/HTTP2 protocols. If using an external load balancer, you may need to explicitly enable these protocols on your load balancers. {{% /note %}}

Set up your object store

InfluxDB Clustered supports AWS S3 or S3-compatible storage (including Google Cloud Storage, Azure Blob Storage, and MinIO) for storing InfluxDB Parquet files. Refer to your object storage provider's documentation for information about setting up an object store:

{{% caption %}} * This list does not represent all S3-compatible object stores that work with InfluxDB Clustered. Other S3-compatible object stores should work as well. {{% /caption %}}

{{% note %}}

Object storage recommendations

We strongly recommend the following:

  • Enable object versioning

    Enable object versioning in your object store. Refer to your object storage provider's documentation for information about enabling object versioning.

  • Run the object store in a separate namespace or outside of Kubernetes

    Run the Object store in a separate namespace from InfluxDB or external to Kubernetes entirely. Doing so makes management of the InfluxDB cluster easier and helps to prevent accidental data loss. While deploying everything in the same namespace is possible, we do not recommend it for production environments. {{% /note %}}

Configure object storage permissions

Ensure the identity you use to connect to your S3-compatible object store has the correct permissions to allow InfluxDB to perform all the actions it needs to.

{{< expand-wrapper >}} {{% expand "View example AWS S3 access policy" %}}

The IAM role that you use to access AWS S3 should have the following policy:

{{% code-placeholders "S3_BUCKET_NAME" %}}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:PutObjectAcl",
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*",
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::S3_BUCKET_NAME",
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*",
        }
    ]
}

{{% /code-placeholders %}}

Replace the following:

  • {{% code-placeholder-key %}}S3_BUCKET_NAME{{% /code-placeholder-key %}}: Name of your AWS S3 bucket

{{% /expand %}}

{{% expand "View the requirements for Google Cloud Storage" %}}

To use Google Cloud Storage (GCS) as your object store, your IAM principal should be granted the roles/storage.objectUser role. For example, if using Google Service Accounts:

{{% code-placeholders "GCP_SERVICE_ACCOUNT|GCP_BUCKET" %}}

gcloud storage buckets add-iam-policy-binding \
    gs://GCP_BUCKET \
    --member="serviceAccount:GCP_SERVICE_ACCOUNT" \
    --role="roles/storage.objectUser"

{{% /code-placeholders %}}

Replace the following:

  • {{% code-placeholder-key %}}GCP_SERVICE_ACCOUNT{{% /code-placeholder-key %}}: Google Service Account name
  • {{% code-placeholder-key %}}GCP_BUCKET{{% /code-placeholder-key %}}: GCS bucket name

{{% /expand %}}

{{% expand "View the requirements for Azure Blob Storage" %}}

To use Azure Blob Storage as your object store, your service principal should be granted the Storage Blob Data Contributor role. This is a built-in role for Azure which encompasses common permissions. You can assign it using the following command:

{{% code-placeholders "PRINCIPAL|AZURE_SUBSCRIPTION|AZURE_RESOURCE_GROUP|AZURE_STORAGE_ACCOUNT|AZURE_STORAGE_CONTAINER" %}}

az role assignment create \
    --role "Storage Blob Data Contributor" \
    --assignee PRINCIPAL \
    --scope "/subscriptions/AZURE_SUBSCRIPTION/resourceGroups/AZURE_RESOURCE_GROUP/providers/Microsoft.Storage/storageAccounts/AZURE_STORAGE_ACCOUNT/blobServices/default/containers/AZURE_STORAGE_CONTAINER"

{{% /code-placeholders %}}

Replace the following:

  • {{% code-placeholder-key %}}PRINCIPAL{{% /code-placeholder-key %}}: A user, group, or service principal that the role should be assigned to
  • {{% code-placeholder-key %}}AZURE_SUBSCRIPTION{{% /code-placeholder-key %}}: Your Azure subscription
  • {{% code-placeholder-key %}}AZURE_RESOURCE_GROUP{{% /code-placeholder-key %}}: The resource group that your Azure Blob storage account belongs to
  • {{% code-placeholder-key %}}AZURE_STORAGE_ACCOUNT{{% /code-placeholder-key %}}: Azure Blob storage account name
  • {{% code-placeholder-key %}}AZURE_STORAGE_CONTAINER{{% /code-placeholder-key %}}: Container name in your Azure Blob storage account

{{% /expand %}}

{{< /expand-wrapper >}}

{{% note %}} To configure permissions with MinIO, use the example AWS access policy. {{% /note %}}

Set up your PostgreSQL-compatible database

The InfluxDB Catalog that stores metadata related to your time series data requires a PostgreSQL or PostgreSQL-compatible database (AWS Aurora, hosted PostgreSQL, etc.). The process for installing and setting up your PostgreSQL-compatible database depends on the database and database provider you use. Refer to your database's or provider's documentation for setting up your PostgreSQL-compatible database.

PostgreSQL-compatible database requirements

  • PostgreSQL version 13 or 14.
  • Minimum of 4 GB of memory or equivalent provider-specific units.
  • To avoid conflicts and prevent issues caused by shared usage with other applications, ensure that your PostgreSQL-compatible instance is dedicated exclusively to InfluxDB.

{{% note %}} We strongly recommended running the PostgreSQL-compatible database in a separate namespace from InfluxDB or external to Kubernetes entirely. Doing so makes management of the InfluxDB cluster easier and helps to prevent accidental data loss.

While deploying everything in the same namespace is possible, we do not recommend it for production environments. {{% /note %}}

Set up local or attached storage

The InfluxDB Ingester needs local or attached storage to store the Write-Ahead Log (WAL). The read and write speed of the attached storage affects the write performance of the Ingester, so the faster the storage device, the better your write performance will be. The recommended minimum size of the local storage is 2 gibibytes (2Gi).

Installation and setup of local or attached storage depends on your underlying hardware or cloud provider. Refer to your provider's documentation for information about installing and configuring local storage.

{{< page-nav prev="/influxdb3/clustered/install/set-up-cluster/" prevText="Back" next="/influxdb3/clustered/install/set-up-cluster/configure-cluster" nextText="Configure your cluster" >}}