Hello everyone! In my previous posts, we have seen how we can automate our home Kubernetes cluster using FluxCD and GitOps. Today I would like to show you how we can use Sealed Secrets to manage our secrets securely in the Git repository. At the same time, the data is stored securely.
What are Sealed Secrets?
Sealed Secrets is a tool developed by Bitnami that allows us to securely encrypt Kubernetes secrets before uploading them to our Git repository. Unlike normal secrets, which are in plain text and can therefore pose a security risk, Sealed Secrets are encrypted and can only be decrypted by the Sealed Secrets Controller in your cluster.
Why Sealed Secrets?
Managing secrets in a public or private Git repository poses a major security risk if they are stored in plain text. Sealed Secrets solve this problem by adding an additional layer of security:
- On-site encryption: Secrets are encrypted on your own device before they are pushed.
- Cluster-specific keys: Each Kubernetes cluster has its own unique encryption key for Sealed Secrets, which further increases security.
Installation of the Sealed Secrets Controller via GitOps
To install the Sealed Secrets Controller with GitOps, we use FluxCD to automatically apply the Kubernetes manifests from our Git repository. Here are the steps:
- Create a new namespace under
clustername/bootstrap/namespace
apiVersion: v1
kind: Namespace
metadata:
name: sealed-secretsCode language: PHP (php)- Add the Sealed Secrets Helm repository in
clustername/bootstrap/helmrepositoryto your Git repository:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: sealed-secrets
namespace: default
spec:
interval: 15m
url: https://bitnami-labs.github.io/sealed-secretsCode language: JavaScript (javascript)- Create a customisation that refers to a helmet release
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: sealed-secrets
namespace: default
spec:
interval: 15m
path: clustername/apps/sealed-secrets
prune: true # remove any elements later removed from the above path
timeout: 2m # if not set, this defaults to interval duration, which is 1h
sourceRef:
kind: GitRepository
name: flux-system
namespace: flux-system
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: sealed-secrets
namespace: sealed-secrets
Code language: PHP (php)- Configure the HelmRelease in
clustername/apps/sealed-secrets. I usually copy the entire supplied values.yaml of a Helm chart. However, I comment out tags for images, for example. These should be controlled by the Helmchart.
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: sealed-secrets
namespace: sealed-secrets
spec:
chart:
spec:
chart: sealed-secrets
version: 2.x
sourceRef:
kind: HelmRepository
name: sealed-secrets
namespace: default
interval: 15m
timeout: 5m
releaseName: sealed-secrets
values:
global:
imageRegistry: ""
imagePullSecrets: []
storageClass: ""
kubeVersion: ""
nameOverride: ""
fullnameOverride: ""
namespaceOverride: ""
commonLabels: {}
commonAnnotations: {}
clusterDomain: cluster.local
extraDeploy: []
image:
registry: docker.io
repository: bitnami/sealed-secrets-controller
# tag: 0.26.0-debian-12-r2
digest: ""
# falls das Keypair in einem eigenen Secret abgelegt
# wird gibt man hier den Namen an
secretName: "name-of-the-secret-containing-your-keys"
.........
- Commit the changes to your Git repository and push them: Flux will automatically recognise the changes and install the Sealed Secrets Controller in the cluster.
Create and manage sealed secrets
To create a sealed secret, we use the tool kubeseal. Here is an example of how to create a secret and seal it securely:
- Create a normal Kubernetes Secret:
kubectl create secret generic mysecret --namespace=namespace --from-literal=password='secret' --dry-run=client -o yaml > mysecret.yaml - Sealing the secret:
kubeseal --cert path/to/your/cert < mysecret.yaml > mysealedsecret.yaml - Add the sealed secret to your Git repository and push the changes: Flux will automatically apply the sealed secret to the cluster.
Best practices for sealed secrets
- Regular key rotation: To increase security, the encryption key of the Sealed Secrets Controller is regularly renewed. The old keys remain valid. The concept is described in detail in the documentation.
- Backup the keys: Make sure you have backups of the private keys so that you can decrypt sealed secrets in an emergency.
Securing and restoring keys with Sealed Secrets
One of the challenges of using Sealed Secrets is handling the key pairs, especially in scenarios such as cluster reinstallations or disaster recovery. Here are the steps on how to secure the keys and restore them if necessary:
Securing the private and public key
- Securing the private key: The private key used to decrypt the sealed secrets is particularly worth protecting. It is stored in the Kubernetes cluster as a secret and should be backed up regularly:
kubectl get secret -n kube-system sealed-secrets-key -o yaml > sealed-secrets-key.yamlThis file should be stored securely, ideally in a location that is both secure and accessible to enable quick recovery in an emergency. - Saving the public key in the Git repository: The public key used to seal new secrets can and should be saved in the Git repository so that developers and CI/CD systems can access it and create new sealed secrets:
kubeseal --fetch-cert > my-cluster-sealed-secrets-cert.pemThis step makes it possible to share the public key without jeopardising security. Save the.pemfile in your Git repository under a path such as./clusters/my-cluster/certs/.
Restoring the keys
When reinstalling the cluster or after a data loss, the key pair must be made known to the system again before any sealed secrets can be used. This is done by restoring the private key from your backup:
kubectl apply -f sealed-secrets-key.yaml
Once the private key has been restored, the Sealed Secrets Controller can continue to decrypt all existing Sealed Secrets without having to recreate them.
Note on new cluster installation
It is important to note that a complete reinstallation of the cluster requires manual intervention to restore the key pair. Without the original private key, all existing sealed secrets are worthless and can no longer be decrypted. Regular backup and careful handling of these keys is therefore crucial for the security and functionality of the cluster.
Conclusion
With Sealed Secrets, there is a secure method of managing Kubernetes secrets in a GitOps environment. By correctly backing up and restoring the keys, we can ensure the integrity and availability of our secret data even in the event of new installations or data loss. This ensures that our cluster can continue to run securely and efficiently without the need to recreate all secrets.
