--- reviewers: - sig-cluster-lifecycle title: Certificate Management with kubeadm content_template: templates/task --- {{% capture overview %}} {{< feature-state for_k8s_version="v1.15" state="stable" >}} Client certificates generated by [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/) expire after 1 year. This page explains how to manage certificate renewals with kubeadm. {{% /capture %}} {{% capture prerequisites %}} You should be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/best-practices/certificates/). {{% /capture %}} {{% capture steps %}} ## Check certificate expiration `check-expiration` can be used to check certificate expiration. ``` kubeadm alpha certs check-expiration ``` The output is similar to this: ``` CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED admin.conf May 15, 2020 13:03 UTC 364d false apiserver May 15, 2020 13:00 UTC 364d false apiserver-etcd-client May 15, 2020 13:00 UTC 364d false apiserver-kubelet-client May 15, 2020 13:00 UTC 364d false controller-manager.conf May 15, 2020 13:03 UTC 364d false etcd-healthcheck-client May 15, 2020 13:00 UTC 364d false etcd-peer May 15, 2020 13:00 UTC 364d false etcd-server May 15, 2020 13:00 UTC 364d false front-proxy-client May 15, 2020 13:00 UTC 364d false scheduler.conf May 15, 2020 13:03 UTC 364d false ``` The command shows expiration/residual time for the client certificates in the `/etc/kubernetes/pki` folder and for the client certificate embedded in the KUBECONFIG files used by kubeadm (`admin.conf`, `controller-manager.conf` and `scheduler.conf`). Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the user should take care of managing certificate renewal manually/using other tools. {{< warning >}} `kubeadm` cannot manage certificates signed by an external CA. {{< /warning >}} {{< note >}} `kubelet.conf` is not included in the list above because kubeadm configures kubelet for automatic certificate renewal. {{< /note >}} {{< warning >}} On nodes created with `kubeadm init`, prior to kubeadm version 1.17, there is a [bug](https://github.com/kubernetes/kubeadm/issues/1753) where you manually have to modify the contents of `kubelet.conf`. After `kubeadm init` finishes, you should update `kubelet.conf` to point to the rotated kubelet client certificates, by replacing `client-certificate-data` and `client-key-data` with: ```yaml client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem ``` {{< /warning >}} ## Automatic certificate renewal `kubeadm` renews all the certificates during control plane [upgrade](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/). This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure. {{< note >}} It is a best practice to upgrade your cluster frequently in order to stay secure. {{< /note >}} If you have more complex requirements for certificate renewal, you can opt out from the default behavior by passing `--certificate-renewal=false` to `kubeadm upgrade apply` or to `kubeadm upgrade node`. {{< warning >}} Prior to kubeadm version 1.17 there is a [bug](https://github.com/kubernetes/kubeadm/issues/1818) where the default value for `--certificate-renewal` is `false` for the `kubeadm upgrade node` command. In that case, you should explicitly set `--certificate-renewal=true`. {{< /warning >}} ## Manual certificate renewal You can renew your certificates manually at any time with the `kubeadm alpha certs renew` command. This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in `/etc/kubernetes/pki`. {{< warning >}} If you are running an HA cluster, this command needs to be executed on all the control-plane nodes. {{< /warning >}} {{< note >}} `alpha certs renew` uses the existing certificates as the authoritative source for attributes (Common Name, Organization, SAN, etc.) instead of the kubeadm-config ConfigMap. It is strongly recommended to keep them both in sync. {{< /note >}} `kubeadm alpha certs renew` provides the following options: The Kubernetes certificates normally reach their expiration date after one year. - `--csr-only` can be used to renew certificates with an external CA by generating certificate signing requests (without actually renewing certificates in place); see next paragraph for more information. - It's also possible to renew a single certificate instead of all. ## Renew certificates with the Kubernetes certificates API This section provide more details about how to execute manual certificate renewal using the Kubernetes certificates API. {{< caution >}} These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead. {{< /caution >}} ### Set up a signer The Kubernetes Certificate Authority does not work out of the box. You can configure an external signer such as [cert-manager][cert-manager-issuer], or you can use the built-in signer. The built-in signer is part of [`kube-controller-manager`][kcm]. To activate the built-in signer, you pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` arguments. The built-in signer is part of [`kube-controller-manager`][kcm]. To activate the built-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags. If you're creating a new cluster, you can use a kubeadm [configuration file][config]: ```yaml apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration controllerManager: extraArgs: cluster-signing-cert-file: /etc/kubernetes/pki/ca.crt cluster-signing-key-file: /etc/kubernetes/pki/ca.key ``` [cert-manager-issuer]: https://docs.cert-manager.io/en/latest/tasks/issuers/setup-ca.html [kcm]: /docs/reference/command-line-tools-reference/kube-controller-manager/ [config]: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 ### Create certificate signing requests (CSR) You can create the certificate signing requests for the Kubernetes certificates API with `kubeadm alpha certs renew --use-api`. If you set up an external signer such as [cert-manager][cert-manager], certificate signing requests (CSRs) are automatically approved. Otherwise, you must manually approve certificates with the [`kubectl certificate`][certs] command. The following kubeadm command outputs the name of the certificate to approve, then blocks and waits for approval to occur: ```shell sudo kubeadm alpha certs renew apiserver --use-api & ``` The output is similar to this: ``` [1] 2890 [certs] certificate request "kubeadm-cert-kube-apiserver-ld526" created ``` ### Approve certificate signing requests (CSR) If you set up an external signer, certificate signing requests (CSRs) are automatically approved. Otherwise, you must manually approve certificates with the [`kubectl certificate`][certs] command. e.g. ```shell kubectl certificate approve kubeadm-cert-kube-apiserver-ld526 ``` The output is similar to this: ```shell certificatesigningrequest.certificates.k8s.io/kubeadm-cert-kube-apiserver-ld526 approved ``` You can view a list of pending certificates with `kubectl get csr`. ## Renew certificates with external CA This section provide more details about how to execute manual certificate renewal using an external CA. To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs). A CSR represents a request to a CA for a signed certificate for a client. In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR. ### Create certificate signing requests (CSR) You can pass in a directory with `--csr-dir` to output the CSRs to the specified location. If `--csr-dir` is not specified, the default certificate directory (`/etc/kubernetes/pki`) is used. Both the CSR and the accompanying private key are given in the output. After a certificate is signed, the certificate and the private key must be copied to the PKI directory (by default `/etc/kubernetes/pki`). A CSR represents a request to a CA for a signed certificate for a client. You can create certificate signing requests with `kubeadm alpha certs renew --csr-only`. Both the CSR and the accompanying private key are given in the output; you can pass in a directory with `--csr-dir` to output the CSRs to the specified location. Certificates can be renewed with `kubeadm alpha certs renew --csr-only`. As with `kubeadm init`, an output directory can be specified with the `--csr-dir` flag. To use the new certificates, copy the signed certificate and private key into the PKI directory (by default `/etc/kubernetes/pki`) A CSR contains a certificate's name, domains, and IPs, but it does not specify usages. It is the responsibility of the CA to specify [the correct cert usages][cert-table] when issuing a certificate. * In `openssl` this is done with the [`openssl ca` command][openssl-ca]. * In `cfssl` you specify [usages in the config file][cfssl-usages] After a certificate is signed using your preferred method, the certificate and the private key must be copied to the PKI directory (by default `/etc/kubernetes/pki`). [cert-manager]: https://github.com/jetstack/cert-manager [openssl-ca]: https://superuser.com/questions/738612/openssl-ca-keyusage-extension [cfssl-usages]: https://github.com/cloudflare/cfssl/blob/master/doc/cmd/cfssl.txt#L170 [certs]: /docs/setup/best-practices/certificates/ [cert-cas]: /docs/setup/best-practices/certificates/#single-root-ca [cert-table]: /docs/setup/best-practices/certificates/#all-certificates {{% /capture %}}