Managing Secrets Safely in Kubernetes

A database password in a Helm chart, an API token in the CI pipeline, and a cloud key as an environment variable: This is how many security problems in Kubernetes begin. Managing Secrets in Kubernetes therefore means not just writing values into a Kubernetes object. It is an operational task that connects access rights, deployment processes, rotation, auditability, and incident response.

For medium-sized businesses, this is not an academic security question. If credentials end up in a Git repository or too many workloads gain access to production keys, concrete risks arise: unauthorized data access, blocked releases, costly incident response, and in the worst-case scenario, a reportable security incident. A viable solution must enable teams to deliver faster without delegating security to manual individual steps.

What Kubernetes Secrets do - and what they don't

A Kubernetes Secret stores sensitive configuration values like passwords, certificates, tokens, or private keys. Pods can mount these values as a file or receive them as an environment variable. This is practical, but it does not create secure secret management.

The critical point: Base64 is not encryption. A standard Secret is merely encoded. Anyone with access to the Kubernetes API, a backup, or a manifest file containing this information can read the value with minimal effort. Additionally, encryption in etcd storage must be explicitly configured. Without Encryption at Rest, secrets may be stored in plain text.

Kubernetes Secrets thus serve as a sensible handover mechanism to applications but are rarely the right sole vault for long-term, highly critical credentials. What architecture makes sense depends on the protection needs, the cloud environment, existing identity systems, and the number of teams involved. For a small, clearly defined cluster, well-encrypted Kubernetes Secrets may be sufficient. In cases of multiple environments, frequent rotations, or regulatory requirements, a dedicated Secret Manager is often the more robust choice.

Managing Secrets in Kubernetes: The Security Foundation

The foundation consists of a few measures that must work together. Individual settings do not solve the problem.

First, Secrets should be stored encrypted in the cluster. This includes an Encryption-at-Rest configuration for the Kubernetes control plane as well as a well-managed key for encryption. In managed Kubernetes offerings, parts of this are often preconfigured. Nevertheless, it must be verified which key is used, who is allowed to administer it, and whether existing Secrets have also been re-encrypted after the feature is activated.

Equally central is RBAC. Access to Secrets should not be granted broadly to developer groups, CI systems, or service accounts. A namespace alone is not a security boundary if roles are overly broadly defined. A deployment typically requires access only to the Secrets of its service account - not to all Secrets in a namespace and certainly not cluster-wide.

Access rights like `list` and `watch` deserve special attention. They appear less harmful at first glance than `get`, but may disclose metadata and, in some configurations, more information than necessary for operation. Rights should be limited to specific resources, namespaces, and workloads. Regular audits prevent historical exceptions from becoming lasting risks.

Logs and diagnostic tools are also part of the security model. Debug outputs, failed pipeline jobs, or support bundles must not contain secret values. This applies not only to Kubernetes itself but also to applications, sidecars, and observability agents. A token in a central logging system is just as problematic as a token in the repository.

No Plain Text Secrets in Git

GitOps enables reproducible deployments, but a Git repository is not a secret store. A checked-in password often remains in commits, forks, local clones, and backups—even if the file is deleted later. This also applies to internal repositories.

For declarative deployments, there are two established approaches. The first approach encrypts Secret manifests before committing. Tools like SOPS or Sealed Secrets allow encrypted values to be version-controlled, which are only decrypted in the target cluster. This makes sense for teams that use Git as a central source for infrastructure changes and manage a manageable number of Secrets.

The second approach references external Secrets from the deployment. A controller reads the values at runtime from a dedicated Secret Manager and creates or updates the required Kubernetes Secrets. This keeps the actual values outside of Git and often also outside of daily Kubernetes administration.

Both models have their place. Encrypted manifests are transparent and well-documented but require a reliable process for key management and rotation. External Secret Managers reduce the spread of sensitive values and support dynamic credentials but introduce an additional dependency into the platform. What matters is that the chosen path is automated, documented, and manageable in case of disruptions.

Identities Instead of Long-Lived Credentials

The strongest lever often lies not in better storage of a password but in avoiding passwords altogether. Applications that access cloud resources should, where possible, be authenticated via Workload Identities. In this approach, a Kubernetes Service Account is assigned a clearly defined cloud identity. The application can access resources, for example, object storage, messaging, or a database service, without carrying a long-lived cloud key in the container.

This significantly reduces the number of critical Secrets. Furthermore, permissions can be linked more precisely to workloads and centrally revoked. For productive platforms, this is usually safer and operationally simpler than a shared API key that remains unchanged over years and across multiple deployments.

For databases, the trade-offs are more nuanced. Not every service supports ephemeral or identity-based credentials. Where dynamic database credentials are possible, they should be preferred. Otherwise, a traceable rotation process is necessary, wherein the application and database can be controlled to switch to new values. Manually changing a password every 90 days sounds compliant but can cause outages if the transition is not automated.

Rotation Must Be a Planned Operational Process

Rotating a Secret is more than just changing a value in the Secret Manager. Applications read environment variables only at startup. If a Secret is provided this way, the pod typically requires a restart for the new value to take effect. Secrets mounted as files can be updated, but even then, the application must detect changes and properly rebuild its connection.

For business-critical systems, rotation should be tested like a deployment. This includes a defined responsible party, an automated process, monitoring for authentication errors, and a fallback plan. For databases, a transitional phase with two valid credentials may be necessary: The new password is rolled out, the application switches, and then the old password is deactivated.

Certificates deserve their own consideration. Their lifetimes and renewals must be monitored because an expired certificate often only becomes noticeable when a connection fails. Automated issuance and renewal significantly reduce this risk but do not replace the monitoring of the actual state.

CI/CD Pipelines as Part of the Attack Surface

The pipeline often receives broader rights than an individual developer. It builds images, deploys in clusters, and accesses artifacts or cloud resources. A compromised pipeline token can therefore jeopardize a large portion of the platform.

The pipeline should use ephemeral identities, not work with permanently stored cluster admin credentials, and only deploy to intended environments. Production releases may require additional controls, but manual copy-and-paste processes are not a security architecture. Better are traceable approvals in the deployment workflow and separate permissions for build, test, and production.

Container images must not contain Secrets either. This sounds obvious, but it regularly occurs through build arguments, configuration files, and multi-stage builds. An image scan helps but does not replace a clear rule: Sensitive values are injected at runtime and never stored in images, artifacts, or test data.

Check, Practice, Adjust

A functioning secret strategy is measurable. Teams should regularly check which Secrets exist, who can read them, when they were last rotated, and whether unused values can be removed. Audit logs from the Kubernetes API and the Secret Manager make accesses traceable. This data is particularly valuable when an employee leaves, a token is accidentally exposed, or a security incident is investigated.

A realistic incident scenario is also part of this: How is a compromised Secret identified, revoked, replaced, and checked against affected systems? Those who test these processes once under controlled conditions avoid hasty decisions in the event of an actual incident. devRocks connects such security mechanisms with CI/CD, observability, and ongoing platform operations so that protective measures do not slow down releases.

Good secret management is not evidenced by the fact that no one sees passwords anymore. It is evidenced by applications accessing only the resources they need in a controlled manner - and that a compromised value can be replaced without long operational interruptions.