Securely managing sensitive data with HashiCorp Vault

As the central infrastructure team, we at Integration Services provide applications for data exchange, among other things. These applications are used not only by OTTO tech teams, but also by teams at the group companies. Our goal is to make the exchange of messages professional, highly available, and as simple as possible. We have been providing Hashicorp Vault for the exchange of sensitive data since 2017. This product is in great demand, and is meanwhile being used by some 270 clients. Each team receives an isolated area in Vault, where data can be deposited. Special application cases also deal with the option of sharing secrets among teams.

So-called secret engines are used to store the secrets. The most widely used are the key-value backends, where you can deposit key-value pairs. There also is the possibility to have dynamic secrets generated, for example OTPs (one-time passwords) that certainly have been used by everyone for two-factor authentication sometime or other, or dynamic AWS IAM users for access to cloud resources.
Vault access is likewise implemented in many ways. For example, personal users can be authenticated with the OpenID Connect (OIDC) method, or technical users with the Vault-internal AppRole and other methods.

But what does secrets administration do for me if I still have to save login credentials for access? In this article, this problem is solved by means of a bridge of trust between the Google Cloud Platform (GCP) and Vault.

Of course, we need not reinvent the wheel, and we can use existing libraries to access Vault. Especially for developments with Java and Spring Boot, the Spring Cloud Vault library suggests itself. As is the Spring fashion, the library is configured solely by means of the property files. Among other things, the Vault URL, and also the authentication and the secret store, are addressed.

A reference implementation can be found on GitHub.

As can be gathered from the configuration, GCP IAM is used here for authentication to Vault, and a GCP service account mail address is handed over. For this to work, the GCP authentication method needs to be activated and configured in Vault first.

Vault and the client each need their own GCP service account. The authorizations iam.serviceAccounts.get and iam.serviceAccountKeys.get are required for Vault, and the roles/iam.serviceAccountTokenCreator authorization for the client. Vault's service account then checks whether the client has authorization for the Vault backend, and allows access, or not.

In the Spring Boot application, the client's service account is entered in the Spring properties. To run the application, e.g. in GCP, you can make it runnable with Cloud Run. Upon starting, the connection is established, and the client is authenticated. Further login credentials are not required, you only find the email address of the service account in the configuration.

Since Cloud Run is linked to GCP's meta data service, an access token can be generated, and the environment variable GOOGLE_APPLICATION_CREDENTIALS can be set already.
It is then used by the Spring Cloud Vault library for authentication to GCP.
To log in to GCP, Vault uses the service account handed over upon the setting-up of the authentication backend. 

The activation of this service account for the client's GCP project creates a bridge of trust between the GCP projects (Vault service account project and client service account project). Vault can check the client's service account, and grant access to it. Vault issues a short-lived token which can be used to read out the secrets.

If the application is started locally, or within an environment outside of GCP, the key of the client service account needs to be provided in a JSON file for the authentication, and the file needs to be linked under the environment variable GOOGLE_APPLICATION_CREDENTIALS.

Due to the integration in Spring, the secrets from Vault can simply be used as properties in the code after successful set-up, in the same way as properties from the application.properties. That is, special calls are not necessary to insert a secret into the code.
Also, when Vault is used, the sensitive data are no longer stored in the key management systems of the cloud, which further improves security, because secrets and application no longer reside in the same project. Using appropriate audit logging alarms and security scans, the Vault environment in our team in addition ensures that the secrets are safe, and cannot fall into the wrong hands.