.Enterprise-scaled Self-Healing StackSets

• Two existing AWS Accounts
• Few AWS StackSets

The following architecture shows the whole solution of the Self Healing StackSets:

Figure 1: Architecture of fully-automated Self Healing Solution with integration to Confluence

The automated generation of the Stackset-configuration via JSON inside the ParameterStore is a multi-purpose-utility:

• Removing the chore to configure manually a JSON-document
• Ensure the Account vending-machines knows what to deploy in which order
• Supporting the self-healing StepFunction about the expected setup of the member-accounts

The lambda responsible for the task is called via an event rule:
Every time a Stackset operation is completed with status "succeeded".
This is due to the fact that the tags of a Stackset are part of the Stackset, not additional items describing a Stackset, so a change to the tags will always result in a Stackset-Update-Operation.

In terms of computer science the Lambda is quite interesting, as the primary problem was to build a nonweighted tree based on the "dependson" and "antidependson" tags and then compile an ordered one-dimensional list, like in the good old "travelling salesmen"-problem.

Figure 2: StepFunction Workflow

ƛ Serverless Functions

• StackSetInitCleanupLambda: Performs a search to identify StackSet instances of AWS Accounts that are either not present within the AWS Organization or deployed to AWS accounts that are suspended. Once identified, proceed with the deletion of these instances from all associated StackSets.
• MandatoryStackSetDeploymentLambda: Search missing StackSets Instances (which are tagged with mandatory = true) and deploy those Instances
• StackSetDriftDetectionLambda: Trigger Drift Detection on all StackSets
• TriggerDriftStatusLambda: Check if Drift Detection is completed on all StackSets
• SearchStackSetInstanceToHealLambda: Searches for drifted StackSet Instances from StackSets which are tagged with Selfhealing = true
• StackSetCleanupLambda: Removes unhealthy StackSet Instances and redeploys them. Parameter Overrides will be cached so the new deployed instance will have the same setting as before.
• StatusPrepareHTMLLambda: Prepare the HTML output Dashboard for Confluence and Json log file of the current StackSet Healthiness State
• TeamsNotificationLambda: Send Teams Notification which summary to notify the GAS Team after each execution

？！Decisions

• InitCleanup Complete: Check whether all unnecessary instances have been removed. If not, StepFunction is triggering the StackSetInitCleanupLambda function again.
• MandatoryStackSetDeployment Complete: Check whether all mandatory instances have been deployed. If not, StepFunction is triggering the MandatoryStackSetDeploymentLambda function again.
• StackSetDriftDetection Complete: Wait until StackSet Drift Detection has been finished on all StackSets
• Healing Complete: Check if all unhealthy Instances are healed otherwise invoke StackSetCleanupLambda again

While developing the solution we faced several limitations. Here are our findings and solutions for that.

🚨 StackSets instance operations: Maximum number of stack instances, across all stack sets, that you can run operations on in each region at the same time, per administrator account is limited to 10.000 operations.

✅ We implemented a counter to count the current StackSets operations which are in progress, in addition we also caught the Exception from CloudFormation and waited few seconds to try the operation again.

🚨 Parameter Overwrites Caching: Whenever removing a drifted StackSet Instance which has Parameter Overwrite you will lose the individual parameters of the instance.

✅ Before deleting the drifted StackSet instance, we cache the parameter overwrites, and after successful deletion, we re-deploy the StackSet instance with the cached parameter overwrites.

🚨AWS Step Functions Payload size: AWS Step Functions supports payload sizes up to 256KB. For our solution, we need more payloads between states, especially if we want to pass our log or the concurrent parameter overwrites per StackSet.

✅ We store our states in an S3 bucket to pass the state. At the end of the execution, we delete the state from S3 so as not to influence the next Step Function execution with the wrong state.

After each execution of the StackSet Health StepFunction, we aim to notify our GAS team about the actions taken during the previous run. Therefore, we have implemented a Teams notification that includes a status update, a link to the generated dashboard, and a link to the log file.

The following screenshot illustrates an example of a Teams notification. It provides a summary report and directs you to the dashboard and log file for further details.

Figure 3: Status updates via Teams

Our StackSet Health Dashboard is a simple HTML file generated by a Lambda function, stored in S3, and distributed via a CloudFront. You can integrate this dashboard into your Confluence or any other internal wiki. This dashboard is secured via the CloudFormation function - in addition, you can also add a firewall to restrict access to a specific CIDR or geographic region and prevent third-party access. The screenshot below shows an example of the overall StackSet Health status information for an entire AWS organization.

Figure 4: Dashboard