Infrastructure as Code (IaC) – keeping Terraform configuration DRY with Terragrunt

We split our GCP projects by their corresponding environment: production, staging, development. The development and staging projects mirror all resources we use for production purposes, though their sizing might differ to reduce overall cloud costs. Usually, a set of resources for these projects includes virtual machines, serverless code execution environments, databases, storage buckets or monitoring configuration.

In addition we also use a separate infra project for basic infrastructure components shared across all environments, like a common Artifact Registry and Cloud Build setup for building and storing Docker images, as well as central resources needed for all projects, e.g. Jenkins or Grafana VMs, which interact with resources in all other environments. The configuration for this project deviates from the rest of our setup.

Lastly, all projects use common and mostly similarly arranged basic components such as network, security, and identity-access configuration. The respective remote state for each environment is kept in a Terraform state bucket within the infra project.

While we keep resources for regularly deployed microservices close to the actual service repository (or at least in a separate repository shared by all microservices with similar deployment mechanisms), there are also a lot of cloud components which are set up and changed infrequently. This basic infrastructure is maintained in a standalone GitHub repository, split up into dedicated folders for each environment, thereby keeping all resources deployed within that project.

At first, this approach was quick to set up and with a limited resource requirement. Maintaining Terraform configuration and running Terraform CLI commands was easy and intuitive.
But with a growing volume of resources, tracking more and more configuration differences between the environments became error-prone with an increasing overwhelmingly and hard- to- maintain file structure (Figure 1).

Figure 1: Basic Terraform file structure without any optimisations

By itself and without further optimizations, this approach revealed several design flaws.

1. Resources were maintained in each project, resulting in repetition of their specification. Changes in the specification had to be carried out for every occurrence, with a high possibility of omitting one or more projects.
2. Provider and backend configuration were also duplicated per project. Although these were mostly identical there was a high chance of misconfiguration – such as an incorrect state file prefix, because Terraform does not allow variable usage within the remote backend configuration.
3. Configuration parameters like sizing variables were statically declared within the project’s config.tf and changing them during Terraform execution was inconvenient and unintuitive.
4. A growing volume of resources within a single Terraform state file increased the execution time of all Terraform CLI commands, because Terraform had to compare each resource in the state file with the actual Cloud resource.
5. Lastly, having a single state file for a large volume of resources severely increased the blast radius if any configuration errors or human mistakes actually occurred. Running an erroneous destroy command on the wrong state file would have been cataclysmic!

Instead, we decided to try Terragrunt, a small wrapper for Terraform, which offers better dynamic remote-state handling and DRY Terraform configuration overall. Terragrunt encourages a modular approach to structuring Terraform code and uses its own configuration files (*.hcl) to keep the resulting code as DRY as possible.

Figure 2: Terraform modules

For migration we first split all resources into logically coherent modules and put them into separate folders within a module directory (Figure 2). Although remote hosting of those modules would have been possible, a local approach seemed sufficient to start with. Simply organizing resources and reworking their configuration with efficient parametrization by creating these modules had a beneficial effect on comprehensible structurization by itself.

While the Terraform configuration and all corresponding *.tf files were kept within each module, the actual environment configuration was condensed within one single terragrunt.hcl file for each module used. The resulting folder structure looked similar to before, while the amount of identical Terraform resource files was dramatically reduced (Figure 3).

Figure 3: Terragrunt environment folders

Figure 4: Terragrunt HCL File including configuration, properties and Terraform functionality

At this step, we could not simply copy an existing file to a new environment or project without making adjustments, since each terragrunt.hcl kept configuration settings unique to its environment. Configuration differences were still scattered across multiple subdirectories.

Terragrunt allows extensive use of variables within its own configuration files, even for remote-state declaration (unlike Terraform). We therefore eliminated the remaining code repetition by using additional *.hcl configuration files that grouped similar properties. The result was terragrunt.hcl files reduced to their actual functionality and completely stripped of any declaration of the properties they use (Figure 5).

Figure 5: Terragrunt HCL file, only including module functionality

Each *.hcl file declaring properties is placed at the level above all folders it relates to. All module and project-related properties for each environment are put into an env.hcl file at the environment level, whereas common environment independent properties are placed within a common.hcl file at the root level. Environment properties for example include the above-mentioned sizing parameters for VMs as well as specific bucket or network names (Figure 6). Common properties at the root level include global configuration parameters, i.e. team labels assigned to each of our resources.

Figure 6: Terragrunt HCL file with environment properties

Furthermore, a single backend and provider configuration is declared within a terragrunt.hcl file at the root level, which is then used by any sub-module in each environment. Since Terragrunt allows the usage of variables in these declarations, the actual code includes placeholders which are propagated during runtime based on the path of each module (Figure 7).

Figure 7: Terragrunt HCL file with dynamic provider and backend configuration

To populate any declared variable, Terragrunt traverses the directories upwards from the module directory the CLI command was run from, including any *.hcl it encounters on the way, until it finds a terragrunt.hcl file at the root level. The wrapper then generates the static Terraform files and executes the corresponding Terraform command (plan, apply, destroy).

The final Terragrunt structure provides a centralizsed declaration of common properties, backend and provider configuration which works for every sub-module and environment without any additional lines of code (Figure 8). All environment-specific properties and configuration is bundled within each env.hcl and can be easily separated from other environments. Therefore, Terragrunt commands run either from our development machines or through automated pipelines do not need to include any variable files or explicitly define any property (although that would be possible).

Figure 8: Final Terragrunt folder structure

Furthermore, modules can easily be added to environments merely by copying the corresponding terragrunt.hcl from any other environment. Don’t need a module anymore? Simply remove its terragrunt.hcl from the environment (after running terragrunt destroy).