Catching a Silent AWS Failure: A Serverless Daily Report for Suspended Auto Scaling Groups
AWS Auto Scaling Groups allow suspending individual processes, a useful feature that can silently cause failures if left suspended. A serverless daily report using EventBridge, Lambda, and SES can surface these issues. The article explains the architecture, deployment steps, common bugs, and lesson…
Amazon EC2 Auto Scaling Groups (ASGs) give operators the ability to pause specific scaling processes—such as Launch, Terminate, HealthCheck, or AZRebalance—during deployments or troubleshooting. While this flexibility is powerful, it also introduces a hidden risk: if a process remains suspended after the incident, the group can silently fail to scale or replace unhealthy instances. The problem is that AWS does not alert you by default when a process is left suspended. You often discover the issue only when traffic spikes or an instance becomes unhealthy, at a time when you need the group to be fully functional.
Why a Daily Check Is Essential
Suspended processes can have immediate operational consequences. A group with the Launch process suspended will not add capacity when demand rises. A group with HealthCheck suspended will not replace instances that fail health checks. These silent failures can cascade into service outages, increased latency, or even data loss. By running a daily check, you surface any suspended processes before they cause problems, giving you a clear view of the group’s health and the potential blast radius of any misconfiguration.
Serverless Architecture Overview
The solution is built entirely on serverless services, keeping costs negligible and maintenance to a minimum. The components are:
- Amazon EventBridge – triggers the Lambda function on a daily cron schedule.
- AWS Lambda – runs a Python script that lists all ASGs in the configured regions, filters for suspended processes, and formats the results into an HTML table.
- Amazon Simple Email Service (SES) – sends the formatted report to a designated email address.
Because the Lambda function is short‑lived and stateless, there are no servers, agents, or cron jobs to maintain. The entire workflow is event‑driven and incurs only the minimal charges for Lambda invocations and SES email delivery.
Core Lambda Logic
The heart of the function performs a paginated scan of all ASGs. A naive single API call would miss groups when the result set is large, so pagination is essential. The code filters for any group that has a non‑empty SuspendedProcesses list and collects key metrics such as desired, minimum, and maximum capacity. The result is then rendered into an HTML table for readability.
Below is a concise version of the logic:
- Instantiate a Boto3 Auto Scaling client for each region.
- Use the paginator for
describe_auto_scaling_groupsto iterate through all groups. - For each group, if
SuspendedProcessesis present, append the group’s details to a list. - Return the list of suspended groups.
Finally, the Lambda function uses SES to send the HTML report. The email includes a toggle—NOTIFY_WHEN_EMPTY—allowing you to receive either a daily all‑clear or only alerts when a problem is detected.
Deployment with the AWS CLI
The deployment is intentionally lightweight, avoiding frameworks or dependencies. A simple Bash script packages the Lambda code and uploads it via the AWS CLI:
- Zip the
handler.pyfile. - Create the Lambda function with the appropriate runtime, role, and handler.
- Set up EventBridge to trigger the function on a daily cron expression.
- Attach an IAM role with the minimal permissions required.
Because the function is serverless, the only ongoing cost is the occasional Lambda invocation and the SES email, both of which are inexpensive for a small number of groups.
Common Pitfalls and Fixes
Two bugs surfaced during the first deployment:
- Runtime.ImportModuleError – caused by a mismatch between the Lambda handler string and the actual file/function name. The handler must be specified as
file.function(e.g.,handler.handler). Updating the configuration withaws lambda update-function-configurationresolved the issue. - AccessDenied for autoscaling:DescribeAutoScalingGroups – the execution role lacked the necessary permission. Adding a minimal inline policy that allows
autoscaling:DescribeAutoScalingGroupsand the SES send actions fixed the problem. Note thatDescribeAutoScalingGroupscannot be scoped to a specific resource, so the policy must grant access to all resources.
These errors are valuable because they point directly to the missing permission or configuration, making troubleshooting straightforward.
Takeaways for Cloud Operators
Suspended ASG processes are a silent risk that can derail deployments if forgotten. A serverless daily check is an inexpensive, low‑maintenance way to surface these issues. By inspecting the error types—runtime configuration errors versus IAM access denials—you can quickly pinpoint and resolve the root cause. Small, automated checks like this can prevent costly downtime and improve overall operational resilience.
Why it matters
Silent suspension of Auto Scaling Group processes can cause unexpected outages during critical periods. A lightweight, serverless daily report ensures these issues are caught early, safeguarding application availability.
Key points
- Suspended ASG processes can silently prevent scaling or instance replacement.
- A serverless daily check using EventBridge, Lambda, and SES is cost‑effective and maintenance‑free.
- Proper pagination is essential when scanning large numbers of ASGs.
- Runtime.ImportModuleError indicates a handler misconfiguration; AccessDenied points to missing IAM permissions.
- A minimal IAM policy with least privilege reduces attack surface and simplifies debugging.
Frequently asked questions
What AWS services are required for this solution?
Amazon EventBridge for scheduling, AWS Lambda for processing, and Amazon SES for email delivery. The solution also uses Boto3 within Lambda to interact with Auto Scaling.
Do I need to pay for this setup?
The Lambda function runs only once per day, and SES email delivery is inexpensive. Overall, the cost is negligible, especially compared to the potential cost of unmonitored scaling failures.
Can I customize the email format?
Yes. The Lambda function builds an HTML table; you can modify the template or add branding as needed.
