Deploying Failure Flags on AWS Lambda

This document will walk you through setting up the Failure Flags agent for Lambda Functions. The Failure Flags agent runs alongside your Function and is responsible for connecting your Function with Gremlin.

Installing the Gremlin-Lambda agent on AWS Lambda

Gremlin-Lambda is a Lambda Extension that you can add to your Lambda Functions. Gremlin-Lambda supports both AMD64/x86_64 and ARM64 architectures. You can learn more about the AWS Lambda Extensions API in the AWS documentation.

Setting required environment variables

You can add the Gremlin-Lambda Extension to any Lambda Function without impacting your function availability or performance. But you do need to add configuration to your environment variables before Gremlin-Lambda will add any value. Configuration comes in via environment variables and or configuration files.

Get started quickly with environment variables only:

1. ^{FAILURE_FLAGS_ENABLED} must be set to either ^true or ^yes or ¹ to enable the Failure Flags SDK in your application.
2. ^{GREMLIN_LAMBDA_ENABLED} must be set to either ^true or ^yes or ¹ to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
3. ^{GREMLIN_TEAM_ID} must be set to your Gremlin Team ID. This and other credential material is available through the Gremlin UI.
4. ^{GREMLIN_TEAM_CERTIFICATE} must be set to your Gremlin Team certificate. Newlines may be preserved using the ^\n escape characters or omitted entirely. This and other credential material is available through the Gremlin UI.
5. ^{GREMLIN_TEAM_PRIVATE_KEY} must be set to your Gremlin Team private key. Newlines may be preserved using the ^\n escape characters or omitted entirely. This and other credential material is available through the Gremlin UI.

If you want to use configuration files, you'll use environment variables to specify the configuration file:

1. ^{FAILURE_FLAGS_ENABLED} must be set to either ^true or ^yes or ¹ to enable the Failure Flags SDK in your application.
2. ^{GREMLIN_LAMBDA_ENABLED} must be set to either ^true or ^yes or ¹ to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
3. ^{GREMLIN_CONFIG_FILE} must be set to the fully-qualified path of a file containing a Gremlin-Lambda configuration file.

Alternatively, you can load configuration files from AWS resources like secretsmanager secrets and ssm parameters:

1. ^{FAILURE_FLAGS_ENABLED} must be set to either ^true or ^yes or ¹ to enable the Failure Flags SDK in your application.
2. ^{GREMLIN_LAMBDA_ENABLED} must be set to either ^true or ^yes or ¹ to enable Gremlin-Lambda. If unset or set to any other value Gremlin-Lambda will operate in NOOP mode.
3. ^{GREMLIN_CONFIG_ARN} must be set to an ARN of a resource containing a Gremlin-Lambda configuration file.
4. ^{GREMLIN_CONFIG_ROLE} must be set to an ARN of a AWS IAM role to use when retrieving the configuration file.

Individual Configuration Values from Files or ARNs

You can configure individual configuration values like ^{GREMLIN_TEAM_CERTIFICATE}, ^{GREMLIN_TEAM_PRIVATE_KEY}, and ^{GREMLIN_CUSTOM_ROOT_CERTIFICATE} to retrieve values from files in your Lambda function or from AWS services using their ARNs. Instead of setting those environment values directly, use their ^_FILE or ^_ARN counterparts. Files must be fully qualified paths from the root of the resulting Lambda Function file system (typically something like ^{/var/task/<path from your code root>}). This project currently supports ^{secretsmanager} secret and ^ssm parameter ARNs.

When you add the Gremlin-Lambda Extension to your Lambda Function and configure the environment variables correctly, your application will be able to consult that extension for Gremlin experiment configuration. You will be able to find your Function in the Gremlin UI under Failure Flags > Services after you launch your app with the layer configured and you exercise the function.

Once you've added Gremlin-Lambda to your project you can use the Failure Flags library (Node, Python, Java, Go) from your Function code!

Creating a Configuration File with Examples

Gremlin-Lambda uses YAML for file-based configuration. These files contain sensitive credential material and should be stored appropriately. The following is a brief description of each property:

1. ^team_id
2. ^{team_certificate}
3. ^{team_private_key}
4. ^labels
5. ^https_proxy
6. ^ssl_cert
7. ^debug
8. ^{api_endpoint_url}

## Gremlin Team Id - you can find this value at https://app.gremlin.com/settings/teams
team_id: ffffffff-ffff-ffff-ffff-ffffffffffff

## Failure Flags Service Labels - Add labels to identify unique deployments.
labels:
    datacenter: corp-na1
    project: columbia

## Debug, set to true for enhanced debug logging to STDOUT
## Uncomment to enable debugging
#debug: true

## Gremlin Team Certificate - Paste certificate content here.
team_certificate: |
    -----BEGIN CERTIFICATE-----
    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXX
    -----END CERTIFICATE-----

## Gremlin Team Certificate - Paste certificate content here.
team_private_key: |
    -----BEGIN EC PRIVATE KEY-----
    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
    -----END EC PRIVATE KEY-----

## HTTPS Proxy, set this when routing outbound Gremlin HTTPS traffic through a proxy
#https_proxy: https://corp.proxy.internal:3128

## Custom CA Certificate, set this when using a https proxy with a self-signed certificate
## Paste certificate content here.
#ssl_cert: |
#    -----BEGIN CERTIFICATE-----
#    ExampleXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#    XXXXXXXX
#    -----END CERTIFICATE-----

‍

Adding the Gremlin-Lambda Extension to your Lambda Function

The Extension's Amazon Resource Name (ARN) varies depending on where your Lambda Function is deployed and which architecture you're using. Retrieve the correct ARN for your architecture and AWS region from this table:

‍

Once you have the correct ARN, add the extension to your Lambda Function by following the instructions in the AWS Lambda Developer Guide.

‍

Building your own Lambda Layer

You may want to build your own Lambda Layer or incorporate the ^{gremlin-lambda} binary in a Lambda container image. Maybe you're at risk of using too many layers, or maybe your organization has a package caching policy. Either way you'll want to build your own asset. You can access all of the ^{gremlin-lambda} and ^{failure-flags-sidecar} builds via our website.

Files provided via assets.gremlin.com have the following format:

^{https://assets.gremlin.com/packages/gremlin-lambda/{VERSION}/{ARCH}/{FILE}}

You can reference the LATEST release at:

^{https://assets.gremlin.com/packages/gremlin-lambda/latest/{ARCH}/{FILE}}

ARCH is one of the following:

• x86_64
• arm64

Different releases may include different files, but the current file set includes:

• gremlin-lambda-linux.tar.gz
• layer-gremlin-lambda.zip
• LICENSE

All versions are listed in a file at: https://assets.gremlin.com/packages/gremlin-lambda/VERSIONS

Debugging Failure Flags

This Lambda Extension provides debug logging when the GREMLIN_DEBUG environment variable is set to true . This information will be included with your Lambda logs and prefixed with [gremlin-lambda] . This additional information will include configuration details, registration status, connection tracing, any relevant errors encountered while interacting with the control plane, and requests for experiments from the Failure Flags SDKs.

‍