Kubernetes Secrets Management: Five Options, Pick the Right One

Cloud Security

Vanilla Kubernetes Secrets are base64-encoded plaintext. Five common alternatives each solve a different problem. The right answer depends on rotation cadence, blast radius, and operator headcount.

By Arjun Raghavan, Security & Systems Lead, BIPI · February 19, 2024 · 8 min read

Vanilla Kubernetes Secrets are base64-encoded, not encrypted (at rest, with the default etcd configuration). Anyone with read access to a namespace's Secret objects has the plaintext. Anyone with read access to etcd has every secret in the cluster. Five common alternatives each solve a different slice of the problem, and we have deployed all of them in client environments. The right answer is rarely the same twice.

Option 1: Native Secrets with etcd encryption

Encrypting etcd at rest with a KMS key (EncryptionConfiguration with kms provider, backed by AWS KMS or GCP KMS) is the baseline. Without it, an attacker with etcd snapshot access reads every secret in the cluster. With it, secrets are encrypted at the etcd layer.

This solves the etcd snapshot problem but not the in-cluster RBAC problem. A pod with permission to get Secrets in its namespace still gets plaintext. For many use cases, this is enough. Add tight RBAC (no list secrets in default service accounts, namespace-scoped) and you have a defensible baseline.

When this is the right answer: small clusters, secrets rotate rarely, the team does not have operational capacity to run more infrastructure.

Option 2: Sealed Secrets

Bitnami Sealed Secrets encrypts secrets with a public key (controller-held private key) so they can be committed to Git. The controller decrypts them in the cluster and creates real Secrets. GitOps friendly.

Limitations: rotation is manual (re-encrypt and commit), the controller's private key is a critical asset, and the in-cluster Secret created by the controller is just a regular Secret with all the same problems as Option 1.

Right answer when: GitOps-heavy team, secrets do not rotate often, no central secrets store.

Option 3: External Secrets Operator (ESO)

ESO syncs secrets from external stores (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, Azure Key Vault, 1Password, and others) into Kubernetes Secret objects. The external store is the source of truth. Rotation happens in the external store and ESO picks up changes on a configurable interval.

This is our default recommendation for most teams. The external store handles versioning, audit logging, access policies, and rotation. ESO is a thin shim that handles the K8s-side delivery. The in-cluster Secret still exists, but it is short-lived and recreated on every external change.

Limitations: still depends on K8s RBAC for in-cluster access. If you need a workload to never see plaintext in a Secret object, ESO is not the answer (use CSI driver). The external store becomes a hard dependency for application starts.

Option 4: Secrets Store CSI Driver

The Secrets Store CSI Driver mounts secrets directly into pod filesystems as files, fetched on pod start from an external store (AWS Secrets Manager, Vault, Azure Key Vault). The secrets never become Kubernetes Secret objects. They never go into etcd.

This is the strongest in-cluster posture. A pod compromise reveals secrets only mounted into that specific pod. Other pods' secrets are not visible. The trade-off is that secrets do not auto-rotate without pod restarts (unless you also enable the optional rotation feature, which has its own quirks).

Right answer when: high-value workloads with strict secret access requirements, willingness to handle pod restarts on rotation, AWS or Azure native preferred.

Option 5: Vault Agent Sidecar

HashiCorp Vault sidecar (or the newer Vault Secrets Operator) injects secrets into pods via a sidecar that authenticates to Vault using the pod's service account token. Secrets are written to a tmpfs shared volume between the sidecar and the application container.

Vault gives the most flexibility: dynamic database credentials (Vault generates a unique DB user per workload, expires it on workload termination), PKI cert issuance, transit encryption as a service. If you need dynamic secrets, Vault is the answer. If you need static secrets only, Vault is overkill.

Cost: Vault is a serious operational commitment. HA Vault with auto-unseal, Raft storage backend, disaster recovery replication, and proper backup is a full-time job. Vault as a Service offerings (HCP Vault, AWS Secrets Manager + Vault namespaces) reduce the operational load but cost more.

Decision matrix

• Small team, AWS, secrets rotate slowly: External Secrets Operator + AWS Secrets Manager.
• GitOps purist, do not want runtime sync: Sealed Secrets.
• High-value workloads, never want plaintext in Secret objects: Secrets Store CSI Driver.
• Need dynamic database credentials, PKI, encryption as a service: Vault.
• Nothing fancy, fine with KMS-encrypted etcd: Native Secrets with tight RBAC.

Common across all options

Regardless of which option, certain things matter: KMS encryption of etcd, deny list verb on Secrets resource for default service accounts, audit logging enabled, secrets never logged by applications, and rotation actually exercised at least quarterly. The mechanism is less important than the discipline around it.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.