Production Secrets Management: Vault, AWS Secrets Manager, Per-Pod Tokens

Cybersecurity

A database password in a Kubernetes secret manifest, base64 encoded, sitting in a git repo, is the configuration we still find in audits in 2023. Real secrets management means short-lived credentials, per-pod identity, and a clear story for rotation. We walk through Vault, AWS Secrets Manager, and the patterns that actually work in production.

By Arjun Raghavan, Security & Systems Lead, BIPI · August 26, 2023 · 10 min read

Secrets management is the security control that everyone agrees is important and almost nobody fully implements. The pattern of a base64-encoded password in a Kubernetes secret is not secrets management. It is obfuscation, and base64 fooled nobody since the day it was specified.

Real secrets management has three properties. The secret is short-lived, the identity that retrieves it is per-workload, and rotation is automated. Anything less is paperwork.

Static secrets are the wrong starting point

If your database password lives forever and is shared across all pods of a service, rotation is a coordinated outage waiting to happen. The fix is dynamic secrets. Vault and AWS RDS can issue per-pod database credentials that expire in an hour. The credential is created on demand, used briefly, and revoked automatically. Rotation becomes a non-event.

• Vault database secrets engine for PostgreSQL, MySQL, MongoDB
• AWS Secrets Manager with rotation Lambda for RDS
• Cloud SQL IAM auth for GCP, no password ever exists
• RDS IAM auth for AWS, the IAM identity is the credential

Per-pod identity is the foundation

Without per-pod identity, you cannot meaningfully limit which workloads can access which secrets. The patterns are well established. AWS IRSA binds an IAM role to a Kubernetes service account. GCP Workload Identity does the same for GCP IAM. Vault Kubernetes auth method binds a Vault policy to a service account. All three give you the same property, the pod identity is verified by the platform, not asserted by the pod.

Once you have per-pod identity, secret access policy becomes a function of the workload, not a function of the cluster. The blast radius of a compromised pod shrinks to the secrets that one pod could read.

If your secrets policy is per-cluster rather than per-pod, a compromise of any pod is a compromise of every secret.

Vault patterns that scale

Vault gives you the most flexibility and the highest operational complexity. For teams that already run it, the patterns are well documented. Vault Agent injector for sidecar-based secret delivery, the Vault Secrets Operator for native Kubernetes integration, and the CSI driver for filesystem mounts.

1. Enable the Kubernetes auth method, configure it against your cluster JWT issuer
2. Define roles that bind service accounts to Vault policies
3. Use dynamic secrets engines where possible, avoid the kv store for anything that supports rotation
4. Inject secrets via Vault Agent or the Secrets Operator, do not bake them into images
5. Monitor Vault audit logs, every secret read is a logged event

AWS Secrets Manager patterns

Secrets Manager is the lower-operational-overhead option if you are AWS-native. Rotation Lambdas, IAM-based access policy, and KMS encryption are all built in. The External Secrets Operator brings Secrets Manager values into Kubernetes as native secret objects, with the IRSA identity controlling access.

The tradeoff is that you are tied to AWS. For multi-cloud or on-premises workloads, Vault is the answer. For pure AWS shops, Secrets Manager plus External Secrets is the cheaper path.

What about config files

Application config that includes secrets should be rendered at startup from a secrets backend, not baked into the image or the helm values. The patterns are well known. Init containers that fetch secrets and write a config file, sidecar agents that maintain a tmpfs file, or direct SDK calls from the application. All of them are better than secrets in environment variables that leak into crash dumps and logs.

Closing

Production secrets management is a platform investment, not an application feature. Build the per-pod identity layer first, pick a secrets backend that fits your operational model, and adopt dynamic secrets wherever the database supports it. The teams that do this once never go back to static passwords in YAML, and the teams that delay it find themselves rotating credentials by hand during every incident.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.