When you investigate secrets management solutions to separate the application credentials from the source code there are a number of topics that should be considered. Each topic should be considered in the context of your environment as they affect your workflow, maintenance load and productivity. Financial industry levels of security does not make sense for most businesses, if they can even implement security to that level.
Encryption
Most solutions to secrets management involve encryption at some point. The entity that controls the encryption controls access to the secrets.
For a hosted service it is imperative that client-side encryption is used with keys that are simply not available to the hosted service. If the service can obtain your keys, your credentials are only secret until your provider gets compromised.
Environment separation
Different execution environments (production, QA, dev, etc) should not have access to each other’s secrets. This will create an opportunity to use the wrong environment’s keys to access a service which can corrupt data or leak production data and cause confidentiality issues. And if a single environment gets compromised, all environments are compromised.
Storage separation
There are some solutions that securely store the secrets in the same repository as the code but in general it is better to store them in different places. It adds another layer of protection.
Access control
Access control is used to limit who has access to the application secrets and the type of access they have, either read-only or read-write access. How finely grained the access levels need to be is a trade-off that must be made. Finer grained control reduces the blast radius when something bad happens but it increases the management effort required for the secrets and it influences what team members are allowed to do. This has an impact on productivity and support. If permissions limit who can deploy to production too strictly it can become a bottleneck in a crisis.
Audit logs
Audit logs can be used pro-actively to prevent security breaches and reactively to investigate security events. These can vary from simple syslog entries for monitoring SSH access to detailed logs that report on every activity happening in the secret management system. The level of logging required is very much dependent on the business environment in which it is used.
Versioning
Secret versioning is necessary to enable zero-downtime secret rotation and secret rollback. Having old versions available may also be useful when investigating security events.
Secret rotation
Your secrets management solution should make it as easy as possible to rotate secrets. You will definitely want to change the secrets as fast as possible after a security event. If the secret management solution makes this difficult it will only add pressure and frustration during an already stressful event.
It is anyway good practice to rotate secrets once every few months and when team membership changes. If the rotation process is easy enough it can happen more frequently and that will reduce the likelihood of compromise. In extreme cases rotation periods are measured in minutes or hours.
To facilitate high-frequency secret rotation the process should not involve downtime.
Rollback
Mistakes happen and sometimes application deployments need to be reverted. The same is true for secret rotations. It should be easy to roll back to an older version of the secret otherwise a small glitch in production will become a major outage.
Secret formats
Secrets come in many forms. Most are simple string values but they can also be JSON objects, SSL certificates or binary files. In fact, it could be anything that can be stored as a disk file. Make sure the solution can deal with the various types of secrets you need to store.
Service availability
The secret management solution’s availability should match your application’s uptime needs. If it becomes unavailable it will have an impact on your service.
When the secrets are provisioned during deployment, unavailability will prevent you from deploying.
When the application actively fetches secrets at runtime, unavailability will cause downtime for your application.
Application deployment process
Secrets can be distributed to the application during deployment or fetched at runtime by the application.
Distribution at deployment has the least impact on the application code because it does not need to be aware of any secret management. The application must be redeployed or restarted when secrets are rotated which may be a problem if rotation happens frequently.
Fetching at runtime requires that the application is aware of the secret management but it enables zero-downtime rotation and very frequent rotation.
Developer productivity
The developers need access to the secrets during development. As new credentials and API keys are added they need to be distributed to the rest of the team to allow them to continue working. The distribution process should be easy and secure otherwise the secrets will get shared via insecure channels like email or instant messaging tools.
Compliance requirements
Regulated industries have defined compliance requirements which dictates the answer to most or all of the mentioned topics.
Comments or questions?
Contact me