SLIM: Hydrating cloud native CI/CD pipelines to securely access GCP projects

So, what would be a situation where such a setup is necessary? Well for example we operate a dedicated Gitlab instance for hosting source code and providing CI/CD capabilities [4]. Our autonomous agile working teams deploy their software products continuously to production using Gitlab CI. So, our centralized CI infrastructure needs high privileged access to various productive GCP projects to deploy IaC and software code.
Our Gitlab instance runs inside a GKE cluster and for each pipeline job a dedicated Gitlab runner pod will be dynamically spawned. From a security standpoint it is challenging to associate an authorized identity to these Gitlab runner pods and to give them access to roll out code in a GCP project.
Especially because this identity needs to be accessible to the workload at any given time as CI jobs can be scheduled at any time.
Our first idea was to create dedicated node pools per team, each with their own GSA. This works but heavily breaks the cloud native paradigm of Kubernetes because job runner pods can’t be scheduled anymore where resources are available but only on a constraint environment. Thus, efforts must be made to make this setup scalable. That’s why we have first started with a trivial solution for this challenge to bootstrap our Gitlab environment and afterwards incorporated our learnings into secret-less-identity-management (SLIM for short) system.

The most straightforward way is to create a unique GSA key per GCP project and store them as a Gitlab CI variable [5]. These variables can be deployed per repository or repository group and needs to be made protected [6], which results in a visual obfuscation in the UI as well as in a restricted accessibility to pipeline jobs running on protected branches and tags.
Obviously, this approach is very easy to implement: We have full admin privileges on our Gitlab instance, so we create a group for each GCP project. Each group belongs to a team; hence we automatically give every team member a Gitlab role to access all repositories inside this group [7]. Then we create for every GCP project a GSA with Owner IAM binding [1] and deploy the GSA key to the corresponding Gitlab group as a CI variable. Making use of these GSA keys is as easy as dumping the content of the variable into a temporary file and activating it via the well-known environment variable GOOGLE_APPLICATION_CREDENTIALS [8] or via gcloud:

Clearly this approach brings several drawbacks: First, there is no automatic rotation mechanism for the GSA keys in place. This implies some security problems, as these keys are valid for 10 years [9]. Thus, we must come up with a solution for this ourselves. Furthermore, the access management to the keys is a nightmare because the accessibility of these high privileged GSA keys is reduced to whether branches and tags are made protected or not. In other words: if anyone manages to commit to a protected branch or create a protected tag, he has full control over everything in the associated Google Cloud project. Additionally, we also had the situation that we have leaked the GSA keys in the Gitlab backup. Making the backup artifacts more security sensitive as they should be.

Going forward, we believe that we can build a better solution for this problem. Our main goal was to get rid of any secrets stored in Gitlab CI variables and simultaneously make it as simple as possible for developers to use. So, we came up with the idea to use the context from where a secret is requested without a shared secret. We also drew some inspiration from the GKE Workload Identity feature [10] as well as the “AWS auth method” of HashiCorps Vault project [11].

A script running inside a CI pipeline needs to access a secret. It authenticates against SLIM service with metadata of its current context. The SLIM service will then validate the metadata and respond with an authentication token. With this token the CI pipeline can fetch a service account key from the SLIM service for a limited time until the token becomes invalid.
This is basically it, making SLIM easy to use and reducing the overall attack surface against SLIM. This is especially important since SLIM needs to have access to all privileged GSA keys requiring high security considerations. But let’s dig deeper on how SLIM validates the context and decides whether a request is trustworthy or not. In figure 2 the architecture and the working principle of SLIM are depicted. Let’s say a CI script needs to access ‘target-project’ via a service account key. For this the runner pod gets its initial GSA using workload identity [10] (1–4). We call this identity the initial context identity. Then the CI script performs a sign JWT request against Cloud IAM [12] with its initial context identity and send this JWT over to the SLIM server with additional metadata from Gitlab CI [13] namely the newly introduced CI_JOB_JWT (5). We call this metadata the context identifier. The SLIM server will first validate the signature of JWT’s with the public key provided by Google [14] and Gitlab [15] (6). Then it adds a database semaphore with the context identifier so that they can’t be reused by any other request. After that it checks with the help of a context provider in our case the Gitlab API if the pipeline job is currently running in the specified repository (7–8). Then it will use a user identity provider, in our case Google Admin Directory, to check if the user who triggered the pipeline job still exists and retrieves the Google groups, the user is member of (9–10). With this information SLIM will use an access provider, in our case a simple Cloud SQL database table, to verify if an associated ACL exists (11). An ACL basically specifies if a resource, the service account key in this case, can be accessed from a user identity or group from a specific context. If this is the case, SLIM will finally use a secret provider, in our case Cloud Secret Manager [16], to retrieve the service account key for ‘target-project’. Before the key is forwarded to the client, an audit provider will be used to audit the whole process. Finally, the runner pod can use the retrieved GSA key to access the ‘target-project’ as usual.

This process may on first glance seem complex, but it basically boils down to two steps: the authentication and the authorization step. In particular, the latter step uses the same mechanics Workload Identity uses internally, that is to validate context metadata.

You may wonder if SLIM is yet another secret manager and why we do not use for example Hashicorps Vault project in the first place. Well, SLIM is actually not a secret manager but rather a secret manager proxy. It’s even possible to use Vault as a backend instead of Cloud Secret Manager of GCP. We rather like to compare SLIM with the Access Proxy in Google’s BeyondCorp architecture [17] because it grants access to a secret based on the identity and the context where the request came from and can thus be more dynamic, a property which typical secret managers usually don’t have.
We built SLIM extensible so that it can be suited for other use cases then only Gitlab CI. These could be accessing secrets from a GKE cluster, a Cloud Function or an App Engine App which simply would need another context provider implementation. How would this look like? For instance, we plan to use SLIM for Kubernetes pods to fetch their credentials to connect to a managed Cloud SQL database. In this dynamic environment where pods come and go, SLIM could use a context provider which uses the pod name, the namespace, cluster name and the GCP project id to check if the pod is actually running and could even decide to only hand out the database credentials on pod initialization and not when the pod is already running.
Internally, we use SLIM daily and for our most crucial CI pipelines where system critical GCP infrastructure is rolled out. We are happy with the system itself and are thinking how we could transfer the concept to more use cases or expand it to new areas. We would appreciate your comments, ideas or suggestions.

This article has been made together with Jan Hicken, Grzegorz Rygielski and Lukas Janssen from Otto Group data.works.