Options to secure application secrets.

It is a generally accepted security principle that application secrets should never be stored as unencrypted text in the source code repository. Some security standards even consider encrypted secrets in code repository as insufficient and demand that the secrets be stored separately from the source. Security conscious developers are then horrified to find their team’s secrets in cleartext inside the repo. Getting those teams to stop their bad habit is usually very difficult.

A big reason people continue to put secrets in the code is because they don’t have an easy-to-use system in place that can effectively deal with secret deployment. Putting such a system in place is a considerable project. The solution must be reliable because it forms part of every application deployment cycle and it must be extremely easy to use otherwise the developers will switch back to putting secrets in code when they are under deadline pressure.

Secret handling systems that are designed to fit in with the team’s existing workflow and which does not break their development momentum will be adopted much quicker than one that forces a new workflow.

Every secrets deployment system is almost by definition a snow flake implementation, even when you use standard software like Hashicorp Vault, because it forms an integral part of your development workflow.

If you are struggling to change your team’s secret handling habits and need to implement a credential management solution, consider the options listed here. Each has advantages and disadvantages. None is suitable for every team or workflow.

Secrets on the app server

Secrets are stored and maintained directly on the application server. The secret files should be kept in a location that is not clobbered by the deployment process.

This the easiest method when you are small but it quickly stops being viable when the number of your applications or servers increase.

Pros:

• It is quick to implement with minimal changes to the application, often only a different path from which to load the secrets.
• No changes to the deployment process.

Cons:

• Access control to the secret files is implemented by SSH and the file permission system.
• No change history.
• No easy rollback.
• It is not scalable.

Secrets encrypted in the source repo

Secrets are stored as encrypted files in the source code repository. The decryption key must not be committed to version control and must still be deployed to the server. Deploying a key is only necessary when the keys change so a manual process like above should be sufficient

Libraries exist to make the decryption process almost transparent for the developers. Some example are git-crypt and Cryptex.

Pros:

• No extra repos or storage locations to maintain.
• Commit comments provide a minimal audit trail about secret changes.
• Roll back of secrets is an option, though a painful one.
• No changes to the deployment process

Cons:

• Access control to secret files is the same as the git repo access.
• No central place for all the team’s secrets. They are spread out across all repos.
• Secret rotation requires re-deployment.

Secrets encrypted in a separate repo

All the team’s secrets are stored in a single, secrets only git repo. The secrets are encrypted.

A library like git-crypt or Cryptex can be used to manage access to the secrets so each application can only decrypt its own secrets.

Pros:

• Only a single additional place to manage for storing all secrets.
• Commit comments provide a minimal audit trail about secret changes.
• Roll back of secrets is an option, though a painful one.
• Secrets are deployed as part of the normal deployment process.
• Finer grained access control is possible. Only a limited number of people need write access to the secrets repo.

Cons:

• Access control is per repo, everybody who can deploy the app needs read access to the secrets repo. Separating project teams requires different secret repos.
• Secret rotation requires re-deployment.

Secrets in a central location

Secrets are stored in a central location where the deployment process has access to them. Part of the deployment process is to fetch the secrets from storage and copy them to the correct location on the application server. Secrets must be encrypted if they are stored in cloud storage or any other off-site location and be decrypted by the deployment process.

This solution works well when there exists a dedicated server from which all deployments are done. In that case the secrets, and access to them, can be managed in a single place.

Implementation is not difficult but it does require changes to every application’s deployment script. The solution is scalable enough to cover most small and medium sized teams’ needs.

Pros:

• Access control on the deploy server is implemented by SSH and the file permission system.
• Access control in cloud storage is enforced by the provider’s solution.
• No extra tools or secret management services required.
• No extra maintenance.
• No extra expenses.
• Secrets are deployed during normal deployment process.
• Central place to manage all secrets.

Cons:

• No audit trail.
• No versioning.
• No roll back.
• Secret rotation requires re-deployment.

Keywhiz

Keywhiz is a self-hosted secrets service for managing and distributing secrets. The Keywhiz server performs the management functions like access control, versioning and auditing. Secrets are accessed using a CLI tool, a JSON API or as a mounted file system.

Keywhiz can run on a single server or scaled up to provide a highly available service when needed. Its simplicity makes it a good option for most businesses which do not need financial-institution levels of security. Keywhiz assumes existence of public key infrastructure (PKI) to identify the clients. The installation instructions provide suggested solutions if no PKI is available yet.

Pros:

• Open source software.
• Relatively simple setup which can scale up.
• No vault unsealing required.
• Three options for requesting secrets: CLI, API and file system mounts.
• Versioned secrets.
• Secret rotation.
• Roll back.
• Fine grained access control.
• Audit trail.

Cons:

• Secrets as a file system mount (FUSE option) has OS specific dependencies.
• Self-hosted service, requires additional maintenance.
• Self-hosted service, has additional server expenses.
• Requires public key infrastructure.

Vault

Vault is a self-hosted secrets management service. It is designed for cases with extreme security and reliability requirements. A vault must first be ‘unsealed’ by providing a decryption key before the server itself gains access to the stored secrets. This unsealing process is designed to require human interaction, ideally multiple persons, at server restart.

Vault is sometimes called the ‘Gold standard’ of secrets management software. This may sound wonderful but the secrecy requirements demanded by financial institutions is overkill for most businesses and may have a downright negative impact on your team’s progress and your application’s reliability. If the Vault server is restarted your application may fail until the vault is unsealed by humans. Auto-unsealing is possible but considered a serious security risk by Hashicorp.

The recommended Vault deployment consists of 8 servers. All these servers need to be maintained in addition to your application.

Pros:

• Open source software.
• Highest security option available.
• Provides many additional security services including an SSH certificate authority.
• Fine grained access control.
• Roll back.
• Secret rotation.
• Audit trail.

Cons:

• Self-hosted service, requires additional maintenance.
• Self-hosted service, high number of servers is expensive.
• Server restart requires human intervention to unseal vault.
• High availability functionality is only available in the enterprise version.

Confidant

Confidant is a self-hosted secret management service for Amazon Web Services. Many of its core features are designed around AWS which prevents its use on other cloud providers.

Confidant can be run from a single server and scaled up to provide a high availability service if needed. Configuration is involved because it includes setting up IAM and KMS. Otherwise it is a relatively simple tool. Secrets are fetched using the CLI client. Confidant should suite most businesses that run their infrastructure on AWS.

Pros:

• Open source software.
• Relatively simple setup which can scale up.
• No vault unsealing required.
• Versioned secrets.
• Secret rotation.
• Roll back.
• Fine grained access control.
• Audit trail.

Cons:

• Self-hosted service, requires additional maintenance.
• Self-hosted service, has additional server expenses.
• Only works on Amazon Web Services.
• Misconfiguration in AWS IAM can cause security holes.

Conjur

Conjur is a self-hosted secrets management service. It is enterprise style software which provides more security services that just secrets management. It has the capability to require human intervention to unseal the vault though it is not the default operation.

A policy-based approach is used for configuring secrets and access to secrets. It is a powerful approach but needlessly complex for smaller teams.

The suggested installation method is to use the official Docker image. Deployment should be relatively easy and it runs from a single server. High availability capability seems to be an enterprise only feature.

Pros:

• Open source software.
• Provides many additional security services.
• Fine grained access control.
• Roll back.
• Secret rotation.
• Audit trail.

Cons:

• Self-hosted service, requires additional maintenance.
• Self-hosted service, has additional server expenses.
• Configuration and use is complex.
• High availability configuration requires the enterprise version.

Chamber

Chamber is a CLI tool for managing secrets in AWS Parameter Store. It depends on AWS for access control and for providing encryption keys. It is not a service but rather a utility that binds various Amazon product into a secrets management solution.

It may be possible to use Chamber on servers outside of AWS if you manage to get them authenticated by AWS. However, Chamber relies on AWS services so you will still need and Amazon account.

Pros:

• Open source software.
• No hosting required.
• Versioned secrets.
• Basic auditing functionality.

Cons:

• Requires AWS account.
• Lot of AWS configuration.
• Access control must be configured in AWS.

Sneaker

Sneaker is a CLI tool for securely storing secrets on S3 using Amazon KMS to provide encryption keys. It is not a service but rather a utility that binds various Amazon product into a secrets management solution.

It may be possible to use Sneaker on servers outside of AWS but you will still need an Amazon account for all the services used by Sneaker.

Pros:

• Open source software
• No hosting required.

Cons:

• Requires AWS account.
• Lot of AWS configuration.
• Secret versioning, access control, logging and auditing are not part of Sneaker. All of them can be provided by Amazon but will need separate configuration.

Cloud provider solutions

Some of the larger cloud providers also offer their own secret management solutions. AWS has Parameter Store which is a free service and Secrets Manager which is a paid-for service. Azure offers Key Vault as a paid-for service. There may be other providers which offer similar solutions.

A particular provider’s solution would generally work best when used inside their own ecosystem. While it may possible to use AWS’ or Azure’s solutions on servers running elsewhere it will likely lead to a setup which is difficult to configure or maintain.

Pros:

• Tight integration into all other products of the same provider.
• Hosted service, no additional maintenance.

Cons:

• Increase provider lock-in.
• Products are generally a building block in a complete secrets management solution. There may be additional configuration needed to get things like auditing or versioning to work.

Solution specific options

Solutions that solve complex problems sometimes provide their own standard way of exposing secrets to the application. Docker and Kubernetes are two such solutions. There is Docker Secrets for Docker and Kubernetes Secrets for Kubernetes. When you need secret management for such a solution the product’s documentation should be your first stop.