As part of our journey towards achieving SOC2 compliance, one of the critical steps we undertook was ensuring the health and performance of our Elastic Load Balancers (ELB) by monitoring their associated target groups. SOC2 compliance is pivotal for us, emphasizing the need to manage and safeguard our customer data effectively.

Normally if you use EKS in combination with a configuration based deployment solutions like we do, then we do not have control over targetgroup names and generated arn. So I decided to automate the configuration of alarms on targetgroup metrics using AWS EventBridge and Lambda.

We are primarily interested when a targetgroup has new targets registered. So we setup an eventbridge rule to generate an event when an ELB has RegisterTarget event generated.

In this blog post, I’ll guide you through the process of setting up AWS CloudWatch alarms for ELB Target Groups using EventBridge and a Lambda function, a task that was essential in our compliance efforts.

Understanding the Need

SOC2 compliance requires stringent monitoring of various aspects of your cloud infrastructure. For our use case, monitoring the health of the ELB Target Groups was crucial. A target group’s health directly impacts the performance and availability of the applications it serves. By setting up alarms, we can be alerted to any issues in real-time, allowing for immediate action to remediate potential problems, thereby ensuring continuous compliance with SOC2’s availability and performance criteria.

The Implementation Journey

The implementation involved three primary components: AWS EventBridge, AWS Lambda, and AWS CloudWatch Alarms. Here’s a step-by-step overview of the process:

1. Setting Up an SNS Topic

First, you’ll need an SNS topic for alarm notifications. This allows you to be promptly alerted when an alarm state is reached.

1. Navigate to the Amazon SNS dashboard in the AWS Management Console.
2. Click on “Topics” then “Create topic”.
3. Choose “Standard” as the type and give your topic a name, such as Prod_CloudWatch_Alarms_Topic.
4. Click “Create topic”.
5. Once the topic is created, create a subscription to get notifications. Click on the topic, then “Create subscription”.
6. Choose a protocol (e.g., Email) and specify the endpoint (e.g., your email address).
7. Click “Create subscription”. You will receive a confirmation email. Confirm your subscription.

2. EventBridge Rule Setup

We started by setting up an EventBridge rule to trigger on specific events related to our ELB Target Groups. The rule was configured to listen for RegisterTargets events, a common action that could impact the health and performance of our target groups.

To trigger your Lambda function in response to specific ELB actions:

1. Go to the Amazon EventBridge console and select “Create rule”.
2. Name your rule and define the event pattern as provided in your initial query.
3. In the Target section, select Lambda function, and choose the function you created.
4. Click “Create”.

The rule pattern used was as follows:

{
  "source": ["aws.elasticloadbalancing"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["elasticloadbalancing.amazonaws.com"],
    "eventName": ["RegisterTargets"]
  }
}

This rule ensures that any time targets are registered (or deregistered) with our target groups, our Lambda function would be invoked to assess and possibly adjust our monitoring setup accordingly.

3. Lambda Function for Dynamic Alarm Management

Your Lambda function will respond to ELB events and manage CloudWatch alarms.

Create Your Lambda Function

1. Navigate to the AWS Lambda dashboard and click “Create function”.
2. Choose “Author from scratch”, give your function a name, and select Node.js as the runtime.
3. Create or choose an existing role with permissions for CloudWatch and ELB access.
4. Click “Create function”.

package.json configuration

Your lambda-function directory should contain a package.json file with the following content:

{
  "name": "lambda-function",
  "version": "1.0.0",
  "description": "AWS Lambda function to manage CloudWatch alarms for Target Groups.",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@aws-sdk/client-cloudwatch": "^3.0.0",
    "@aws-sdk/client-elastic-load-balancing-v2": "^3.0.0",
    "@aws-sdk/client-ssm": "^3.0.0"
  }
}

Our Lambda function, written in Node.js, plays a pivotal role in this setup. It dynamically creates or updates CloudWatch alarms based on the events received from EventBridge. This approach ensures that our alarms are always in sync with the current state of our target groups and load balancers.

The function performs the following actions:

• Extracts the target group ARN from the event and uses it to retrieve the associated load balancer information.
• Constructs the alarm parameters, focusing on the UnHealthyHostCount metric, which is critical for understanding the health of our target groups.
• Uses the AWS SDK for JavaScript (v3) to interact with CloudWatch and create/update the necessary alarms.

index.mjs configuration

Below is a snippet from our Lambda function showing how we construct the CloudWatch alarm:

import { CloudWatchClient, PutMetricAlarmCommand } from "@aws-sdk/client-cloudwatch";
import pkg from '@aws-sdk/client-elastic-load-balancing-v2';
const { ElasticLoadBalancingV2Client, DescribeTargetGroupsCommand } = pkg;

const cloudWatchClient = new CloudWatchClient({ region: "us-east-1" });
const elbv2Client = new ElasticLoadBalancingV2Client({ region: "us-east-1" });

export const handler = async (event) => {
    console.log("Event: ", event);

    try {
        const targetGroupARN = event.detail.requestParameters.targetGroupArn;
        // Extracting the Target Group name (tgName) directly from ARN might not be straightforward as the ARN format
        // might be different. Make sure to adjust the extraction logic based on the actual ARN format you receive.
        const tgName = targetGroupARN.split(':').pop();
        const describeTGCommand = new DescribeTargetGroupsCommand({ TargetGroupArns: [targetGroupARN] });
        const tgResponse = await elbv2Client.send(describeTGCommand);

        if (!tgResponse.TargetGroups || tgResponse.TargetGroups.length === 0) {
            console.log('No Target Groups found.');
            return;
        }

        const loadBalancerArns = tgResponse.TargetGroups[0].LoadBalancerArns;
        if (!loadBalancerArns || loadBalancerArns.length === 0) {
            console.log('No Load Balancers associated with the Target Group.');
            return;
        }

        // Extracting Load Balancer name from ARN might require similar attention to the extraction logic.
        const loadBalancerArn = loadBalancerArns[0];
        var lbName = loadBalancerArn.split(':').pop();
        lbName = lbName.replace('loadbalancer/','');
        
        console.log('LoadBlanacer Id : ' + lbName);

        const alarmParams = {
            AlarmName: `UnhealthyHostCount-${tgName}`,
            ComparisonOperator: 'GreaterThanThreshold',
            EvaluationPeriods: 1,
            MetricName: 'UnHealthyHostCount',
            Namespace: 'AWS/ApplicationELB',
            Period: 300,
            Statistic: 'Average',
            Threshold: 1,
            ActionsEnabled: true,
            AlarmActions: ['arn:aws:sns:us-east-1:123456789:Prod_CloudWatch_Alarms_Topic'], 
            AlarmDescription: 'Alarm when UnhealthyHostCount is above 1',
            Dimensions: [
                {
                    Name: 'TargetGroup',
                    Value: tgName
                },
                {
                    Name: 'LoadBalancer',
                    Value: lbName
                }
            ],
        };

        const putAlarmCommand = new PutMetricAlarmCommand(alarmParams);
        const alarmData = await cloudWatchClient.send(putAlarmCommand);
        console.log("Alarm created/updated: ", alarmData);
    } catch (err) {
        console.error("Error in processing: ", err);
    }
};

4. Deploy Your Lambda Function

Now, deploy your Lambda function by uploading the code through the AWS Management Console or using AWS CLI. Make sure your index.mjs and package.json are included in the deployment package.

5. Testing and Validation

After setting everything up, it’s crucial to test your configuration:

• Trigger a target group change event (e.g., register or deregister targets).
• Verify that your Lambda function executed correctly in the Lambda console’s Monitoring tab.
• Check that the corresponding CloudWatch alarm is created or updated in the CloudWatch console.
• Ensure that you receive a notification through your SNS topic subscription.

Conclusion

Following these steps, you’ve created a robust monitoring solution for your ELB Target Groups, aligning with SOC2 compliance requirements. This setup not only helps in proactive issue resolution but also in maintaining the high availability and performance of your applications, crucial for SOC2’s focus on security and availability.