Why should you care?
Mismanaging secrets can have serious consequences for businesses, including reputation damage, service outages, and even lawsuits from clients. Secret breaches are unfortunately common, and often result from oversights in secrets management rather than sophisticated hacking attacks.
A short search online will reveal a handful of secret breaches from the recent past:
- Credit card number leaked on the dark web amid Australian Clinical Labs cyberattack
- SHEIN fined US$1.9mn over data breach affecting 39 million customers
- Major password manager “LastPass” suffered a breach – again
In order to reduce the risk of a secret breach, it is important to have a proper strategy for managing secrets and to implement best practices in secrets management. This can include using secure methods for storing and accessing secrets, as well as regularly reviewing and updating your secrets management practices. By taking these steps, you can help protect your business and your clients from the risks of secret breaches.
What are secrets?
In our online system we store a lot of data, we can generally classify our data into 3 categories, statistical, private, and secret. Here are the guidelines on how to think about our data:
Statistical
Any data which if leaked is benign to your organization and your users. Statistical data is something your organization might publish on your website or on your marketing material. The data will not contain any user-specific information, nor it’ll be possible to extract user-specific information from the data.
Private
Most of the data you store will end up in this category. This category will contain the private information of your organization, your users’ data, and the data of any 3rd party which operates with your business
Secret
A secret is a digital token or asset that is used for authorization or authentication with any system. Examples of such tokens or assets are:
- Passwords & usernames
- API tokens
- OAuth tokens, Access, and Refresh tokens
- AES keys (symmetrical encryption keys)
- RSA keys (private and public asymmetrical encryption keys)
- SSL Certificates
This is a sensitive layer that requires an added layer of special attention and security. Below we will focus on the proper way to manage these secrets.
Secrets management principles
Any secrets management system we will be using or building should respect the 3 principles of Securing, Storing, and Accessing.
Securing
Secrets should be stored in a secure way, they’ll be encrypted or hashed, and the keys to decrypt the secrets will be stored in a different place. In case of a data breach and someone gains access to your secrets management system, they will only have access to the encrypted secrets and won’t be able to use them.
Storing
A secret management system will take care of the storage of your secrets. A good secret management system will ensure the reliability of your secrets and prevent data loss. Secrets will be stored in multiple copies, and if your system requires, they’ll be stored in multiple copies across multiple clouds or on-prem locations.
Accessing
Secrets should only be accessed by an authorized system or person. Access should be audited. Secrets should only be transferred on a well-configured TLS layer which will encrypt the secrets in transit.
What are the bad options for storing secrets
In the following list, I will outline some bad practices I have encountered in my years of experience. You can compare this list to your organization’s current secrets storage methods to identify potential issues and risks.
Storing secrets in a public repository
Committing secrets to a public repository is a common mistake that can have serious security implications. Malicious actors frequently scan public repositories for secrets, so even if your repository is obscure, it is still at risk of being discovered.
In order to protect sensitive information, it is important to use secure methods for storing and accessing secrets, and to regularly scan your repositories for potential leaks. Tools like TruffleHog and git-secrets can be incorporated into your CI/CD process to help catch leaked secrets.
If you do accidentally commit a secret, it is important to not only delete it from the file, but also delete the relevant commits from your repository’s history in order to remove any trace of the secret.
Storing secrets in a private repository
While storing secrets in a private repository may seem like a secure option (perhaps the word “private” causes confusion), it actually violates important security and access principles. Private repositories are not encrypted, so they may be vulnerable to being accessed by malicious actors.
In addition, anyone who has access to the repository will also have access to the secrets, and copies of the repository may be made for various purposes, further spreading the secrets. This violates the principle of least privilege, as access to secrets is not restricted to only those who need it.
Instead of using private repositories, it is recommended to use one of the secrets management solution described below.
Storing secrets in a container image
Now that we have avoided the common pitfalls of secrets management, we need to consider how to securely inject secrets into our running containers. One tempting solution is to build our container image with the secrets included. However, this carries the risk of improper secret storage, as secrets within the image are not encrypted and can be accessed by anyone with access to the image. Therefore, this is not a recommended approach.
The recommended approach will be to inject the secrets at the container run time as environment variables and avoid inserting the secrets into the image at build time.
Storing secrets on a local machine
Storing secrets locally makes it very convenient when developing. A lot of companies’ production environments are still based on manual deployments where a developer drags and drops a secrets file to a remote machine using the FTP protocol.
The secrets stored on local machines are vulnerable to being picked up by anyone who got access to these machines. This includes ANY software running on the machine and ANY vulnerability that software might be exposed to.
It is discouraged to store secrets on the local machine, and yes, that also means the ~/.aws/credentials file.
Storing secrets on a remote machine
What about a remote machine? In some way, this entire post is about encouraging you to switch to storing your secrets on a remote secret management system. But there is a difference between storing your secrets in plain text on a remote machine and a management system. So if you already made the leap of storing your secrets remotely keep reading to make sure you do it in a proper way.
Storing secrets in a database
In some ways, using a database to store your secrets can be acceptable. Databases have good access management (if properly configured), and databases can be distributed to prevent data loss, BUT databases usually store the data in plain text or in some compressed format.
If you encrypt your secrets inside the database, and you store the encryption key somewhere separate, then databases might be a viable option.
If you are not an encryption expert, and you don’t want to risk it, choose one of the options below and use a dedicated secrets management system.
So what should I do?
Now that we have established the importance of properly managing secrets, it is clear that using a secrets management system is the way forward. Let’s explore our options.
Cloud solutions
If you are working with one of the 3 big clouds, your first option would be one of their key management services:
- For Amazon’s AWS – Secrets Manager
- For Google’s GCP – Secret Manager
- For Microsoft’s Azure – Key Vault
They are great solutions to start with as they integrate with their own container services and Kubernetes. And if your application is more traditional and runs on a server, all 3 clouds provide simple-to-use SDKs to integrate with your app to pull the secrets at runtime.
Although these services will solve most of your issues, they are still limited, and more advanced and flexible solutions exist.
Managed and open-source solutions
A need for more flexible solutions comes up when your environment is deployed across different regions and different cloud providers or if you want to manage your secrets by yourself without being dependent on a specific cloud solution.
In this situation you can rely on the following proven solutions:
Vault by HashiCorp
Vault is often my preferred solution for secret management. It is widely regarded as the industry standard and covers all the key aspects of secret management.
One of the standout features of Vault is its versatile storage backends. If you are deploying on-premises, you can set up a single Vault node to store secrets encrypted locally. Alternatively, if your system is distributed across multiple cloud providers, you can deploy a cluster of Vault nodes and use the Raft algorithm to ensure secrets are kept up-to-date across all nodes.
Vault also supports multiple identity providers, so you can control access to secrets using policies. This makes it easy to grant users and systems only the level of access they need to function.
Keywhiz by Squre
Keywhiz is a solution created for Square that enables the secure storage of encrypted secrets on a MySQL database. Clients use mTLS (mutually authenticated transport layer) to securely retrieve and manage secrets they have access to. Keywhiz is a lightweight solution that can be used on-prem.
Knox by Pinterest
Knox is a solution made by Pinterest. Out-of-the-box Knox only supports an AWS backend with Zookeeper, but Knox has the option of creating custom backends by building on its interfaces.
Confidant by Lyft
Confidant is based on DynamoDB which requires the use of AWS as a backend. Confidant was built for the internal uses of Lyft and it might fit your specific use case.
In Summary
Failing to properly manage secrets can have serious consequences for businesses. To reduce the risk of secret breaches, it is important to have a proper strategy for managing secrets and to implement best practices in secrets management.
Avoid common pitfalls like storing secrets in repositories or on shared or local machines, and instead use a suitable secret management solution to increase your security posture and reduce the risk of secrets being leaked from your organization.
By taking these steps, you can help protect your business and your clients from the risks of secret breaches.
