In this tutorial, you’ll learn how to run a Chaos Engineering experiment on a Python application running on AWS Lambda using Failure Flags. Failure Flags is a Gremlin feature that lets you inject faults into applications and services running on fully managed serverless environments, such as AWS Lambda, Azure Functions, and Google Cloud Functions. With Failure Flags, you can:

• Add latency or errors to applications.
• Inject data into function calls without having to edit or re-deploy source code.
• Simulate partial outages behind API gateways or reverse proxies.
• Customize the behavior and impact of experiments.

This tutorial will focus on testing a Python application on AWS Lambda. You can learn about our other supported languages and platforms in the Failure Flags documentation.

‍

Overview

This tutorial will show you how to:

• Install the Failure Flags Python SDK.
• Deploy an application and the Failure Flags agent to AWS Lambda.
• Run a latency experiment using Failure Flags.

‍

Prerequisites

Before starting this tutorial, you’ll need:

• A Gremlin account (sign up for a free trial here).
• An AWS account with access to Lambda (you can use the lowest-tier x86 or Arm instance for this tutorial to save on costs).
• Python 3.X installed on your local machine (this tutorial was written using Python 3.12.2).

‍

Step 1 - Set up your Python application with Failure Flags

In this step, we’ll create a Python application and add a Failure Flag. This is a simple application that responds to HTTP requests with the current timestamp and the time taken to process the response.

Start by creating a virtual environment for your project, then activate it:

python -m venv venv
source venv/bin/activate

‍

Next, import the required libraries into your app. Lambda expects to find Python dependency folders in the application’s root directory. For the sake of this tutorial, you can do this by redirecting the output of pip install to the current working folder:

pip install --target . -r aws_xray_sdk jsonpickle failureflags

‍

For the application code, create a new file named lambda_function.py with these contents:

import os
import logging
import json
import time
from failureflags import FailureFlag 
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

logger = logging.getLogger()
logger.setLevel(logging.INFO)
patch_all()

def lambda_handler(event, context):
    now = time.gmtime()
    start = time.time()

    # Define the Failure Flag
    FailureFlag('http-ingress-python', {}).invoke()
    end = time.time()
    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json'
        },
        'body': json.dumps({
            'processingTime': round((end*1000) - (start*1000)),
            'timestamp': time.strftime('%Y-%m-%dT%H:%M:%S', now)
        }, sort_keys=True, indent=2)
    }

‍

This application simply responds to incoming requests by printing the time taken to process the request, and the current timestamp. The most important part of this code is line 18. This is where we define our Failure Flag and invoke it if an experiment is running. The first argument is the name of the Failure Flag (in this case, http-ingress-python), and the second is a map of any labels we want to attach to this specific Failure Flag. For example, we could pass the name of the method (lambda_handler) or details about the request. You can leave this empty for now. After the Failure Flag is defined, we calculate how long it took to process the request and return it so it shows in the user’s browser.

‍

Step 2 - Download your client configuration file

Before you can deploy your application, you’ll need to authenticate it with Gremlin. Gremlin provides a downloadable client configuration file that you can use to authenticate any Gremlin agent, including Failure Flags. This file contains your Gremlin team ID and TLS certificates, but you can add additional labels like your application name, version number, region, etc.

1. Download your client configuration file from the Gremlin web app and save it in your project folder as config.yaml.
2. Optionally, add any labels to your configuration file. You can use these labels to identify unique deployments of this application, letting you fine-tune which deployments to impact during experiments. For example, you could add the following block to identify your function as being part of the us-east-2 region and the http-ingress project, letting you target all functions running in us-east-2 or that belong to the http-ingress project:

labels:
    datacenter: us-east-2
    project: http-ingress

‍

The configuration file supports other options, but the defaults are all you need for this tutorial.

‍

Step 3 - Deploy your Python application to Lambda

Next, let’s deploy our application to Lambda.

So far, we’ve configured our application and the Failure Flags SDK. The SDK is what’s responsible for injecting faults into our app, but it doesn’t handle communicating with Gremlin’s backend servers or orchestrating experiments. For that, we need the Failure Flags Lambda layer.

We’ll deploy the Lambda layer alongside our Python app. The specifics of deploying an application to Lambda go beyond the scope of this tutorial, so we’ll link to the AWS docs where necessary.

1. Follow the instructions in Creating a .zip deployment package with dependencies. Remember to download your Gremlin client configuration file to your project folder!
2. Create a new Lambda function using the instructions in Creating and updating functions with .zip files using the console.
3. Before deploying the function, we need to add some environment variables. These are necessary for enabling Failure Flags. In the AWS Console, select the Configuration tab, then select Environment Variables. Click Edit, then enter the following variables:
   1. FAILURE_FLAGS_ENABLED=1
   2. GREMLIN_LAMBDA_ENABLED=1
   3. GREMLIN_CONFIG_FILE=/var/task/config.yaml
4. Optionally, you can provide additional metadata via environment variables, including your Gremlin team credentials. This isn't necessary for this tutorial, since we're authenticating via a config file. See the Failure Flags installation docs for details.
5. Click Test to confirm that your application can receive and process requests correctly.
6. Now we need to add the Failure Flags Lambda layer. Select the Code tab, then scroll down to Layers and click Add a layer:
   1. Under Choose a layer, select Specify an ARN.
   2. Enter one of the ARNs presented in this link, depending on which region and architecture your function is running on. For example, if your Lambda is running in us-east-2 on x86, enter arn:aws:lambda:us-east-2:044815399860:layer:gremlin-lambda-x86_64:13.
   3. Click Verify to confirm that the ARN matches your region and architecture, then click Add.
7. Publish your Lambda by scrolling to the top of the page, clicking Actions, then clicking Publish new version. Enter a name for this version, then click Publish to push your function live.
8. Create a new Function URL by following the instructions in Creating and managing Lambda function URLs. Once the URL is created, click on the link to see your function's output in a new tab. It should look similar to the following:

{
  "processingTime": 880,
  "timestamp": "2024-03-15T16:34:50"
}

‍

Step 4 - Run an experiment

Now for the fun part: running an experiment!

To set the context: Lambda functions are susceptible to changes in network throughput. In other words, if our network connection becomes congested or slow, we’ll likely see less throughput and slower response times. The question is: how does this added latency impact the overall performance of our Lambda function, as well as any other services that depend on it? To test this, we’ll run a latency experiment, add one full second of latency, and observe what happens to our application.

1. In the Gremlin web app, select Failure Flags in the navigation pane (or click this link).
2. Click + Experiment to create a new experiment.
3. Enter a name for the new experiment.
4. Under Failure Flag Selector, click the combo box to show a list of active applications with Failure Flags that Gremlin detected. If your application doesn't show up, confirm that it's finished deploying on Lambda and has responded to at least one request.
5. Optionally, you can add any additional attributes, such as label selectors, in the Attributes box. You can ignore this field for this tutorial.
6. In the Effects box, specify the impact that you want to have on your app. For example, say we want to add 1000 ms (one second) of latency to each call to this function. We can do this by adding the following JSON to this field:
   1. { "latency": 1000 }
7. Set the Impact Probability percentage. For now, set it to 100% to ensure that every call to this function gets impacted.
8. Optionally, change the Experiment Duration to your preferred time. For now, set it to 5 min so you have plenty of time to observe the impact. You can always stop the experiment using Gremlin's Halt button.
9. Click Save & Run to start the experiment.

While the experiment runs, open your Lambda URL in a web browser or a performance testing tool. How is it responding? How noticeable is the latency? Is the amount of latency more than you expected (longer than one second)? If so, why do you think that is? How might you rearchitect this application so the latency doesn't have as big of an impact?

When you're finished making observations and want to stop the experiment, simply click Halt this experiment in the Gremlin web app to stop the experiment.

‍

Conclusion 

Congratulations on running a serverless Chaos Engineering experiment on AWS Lambda with Gremlin! Now that you have Failure Flags set up, try running different kinds of experiments. Add jitter to your network latency, impact a larger or smaller percentage of traffic, generate exceptions, or perform a combination of effects. For more advanced tests, you can even define your own experiments or inject data into your app. Failure Flags also has language-specific features, but this is currently only available for Go.

If you'd like to try Failure Flags outside Lambda, we also have a sidecar for Kubernetes. Just deploy the sidecar, then define and run your experiment. Remember that Failure Flags has no performance or availability impacts on your application when not in use, so don't be afraid to add it to your applications. We also offer SDKs for Node.js, Golang, and Java. These are all available on Github under an Apache-2.0 license.