I have been going down a NixOS journey lately and in doing this have started to work out how to store my secrets that I use in my config in a… secret way?
It started with just putting them in a separate file and including them to the config like environment variables and making sure only root had access to the file/folder.
Then I went down the path of encrypting the files with something like SOPS and then SOPS-Nix to include them in my NixOS config.
From what I can tell that only really lets me safely store them in Git and not much else as if someone was to manage to get root on my server the secrets will still be readable.
I see there is a movement to using KMS in which pass the encrypted information needed and auth to the KMS server and it will pass back the decrypted content. The value here is that you can reject the auth of the server in case of compromise without needing to rekey a whole bunch of files. The other issue with that is that I do not want to use AWS, Azure or GCP as my KMS server. tin foil hat.
I wanted to open up a discussion on the pros and cons of a few of the methods available and if there are any other good ideas which I have sure not thought of.
So from least complex to most complex:
Just put the secrets in root only readable files and leave them on the server and maybe take a backup of them on USB stick on case of server rebuild.
Use something like SOPS to encrypt and SOPS-Nix to add to Nix config allowing me to add configs to a git repo. (issue is if keys get compromised i would need to somehow clear the history in git or change all passwords that that key had access to.)
Look at a KMS server to dynamically decrypt information off server and pass it back.
While I haven’t used NixOS I do use SaltStack for config management of servers, so there is a bit of overlap…
For systems where secrets are only needed to be managed/used locally I generally avoid “storing” the secrets and just generate them as needed. For an example of what I mean, think of a web app that uses a local database. The database password only needs to be know to the web app, so I use SaltStack to generate a random password for the database user and configure the web app to use that password. The password is “cached” locally, but doesn’t need to be known outside the local system (ie. doesn’t need to be stored in a git repo or password management system). If the password is lost for some reason (ie. disaster recovery), a new one is automatically generated and the database user and web app are updated as well. Technically this probably falls into the probably falls into the “root only readable file” category, but has the added benefit of being zero-effort/knowledge management.
For cases where secrets need to known across systems (ie. the web database is on another server) then generating secrets on the fly isn’t really and option and I think it comes down to the type of “team” managing the system, and the environment the system is running in.
Using something like SOPS (or age, GPG encrypted values, etc.) might be fine in some situations, but they do have a few drawbacks:
If the encryption key/passphrase is leaked, then all current and historical secrets are compromised.
Making your configuration manage repo public could be potentially problematic (sure all the secrets are encrypted, but there is still some risk associated with have those encrypted values publicly available).
Doesn’t necessarily scale with with a team size > 1 (do you really want everyone sharing the same key/passphrase, what about when a team members leaves? the only secure option is the generate a new key, rotate all the secrets and encrypt the new values)
Using something like Vault or Infisical might be the best approach, but that requires setting up extra infrastructure and you probably don’t want to be exposing those services publicly, so not necessarily great for use with VPS and the like that exist on potentially hostile networks.
One middle ground I think that could be a potential solution is to fetch secrets from a password manager. A tool like BitWarden allows sharing secrets between accounts, so team members are using their own credentials, instead all sharing the same keys/passphrases/etc. The encrypted secrets don’t need to be stored in the config management repo. The trade-off is that it might require a manual step each time a secret needs to be fetched (ie. type in a passphrase to unlock the vault). This also assumes that bindings between your preferred config management tool and password manager exist.
Most of the stuff I deal with is on-premise (both personally and professionally) so I don’t really want to have to deal with cloud providers for secret management. So I haven’t really looked at KMS and the like.