This document will walk you through setting up Failure Flags for AWS Lambda Functions. On Lambda, Failure Flags is deployed as a Lambda Extension called Gremlin-Lambda. This extension creates a sidecar container that runs alongside your function. Gremlin-Lambda supports both AMD64/x86_64 and ARM64 architectures. You can learn more about the AWS Lambda Extensions API in the AWS documentation.
Configuring Failure Flags for AWS Lambda
The Gremlin-Lambda Extension accepts the following environment variables in addition to those defined in Configuring Failure Flags. When referencing files, the filename must be a fully qualified path from the root of the resulting Lambda Function file system (typically /var/task):
Environment variable
Description
GREMLIN_LAMBDA_ENABLED
Set to true, yes, or 1 to enable the Gremlin Lambda extension.
FAILURE_FLAGS_ENABLED
Set to true, yes, or 1to enable the Failure Flags SDK in your application. If you’re using Failure Flags by proxy, you don't need to set this variable.
GREMLIN_METRICS_EXCLUDE_HEALTH_CHECK_PATHS
A comma-separated list of URL paths that you want to exclude from metrics collection. This is primarily for excluding paths used for Health Checks.
GREMLIN_METRICS_OPT_OUT
Set to true if you do not want the sidecar to collect application metrics (metrics are used for Intelligent Health Checks.
GREMLIN_SIDECAR_ENABLED
Set to true, yes, or 1to enable the Failure Flags sidecar. If this is unset or set to any other value, the sidecar will operate in NOOP mode.
GREMLIN_TEAM_ID
Your Gremlin Team ID. This and other credential details are available in the Gremlin web app.
GREMLIN_TEAM_CERTIFICATE
Set to the contents of your Gremlin Team certificate. This and other credential details are available in the Gremlin web app. Preserve newlines by adding the \nescape character at the end of each line, or by omitting them entirely.
GREMLIN_TEAM_PRIVATE_KEY
Set to the contents or file path of your Gremlin Team private key. This and other credential details are available in the Gremlin web app. Preserve newlines by adding the \nescape character at the end of each line, or by omitting them entirely.
SERVICE_NAME
Set the name that you want to register the service as in Gremlin. Service names can only contain alphanumeric characters, hyphens, and underscores, and must be less than 64 characters long. If not set, Gremlin will auto-detect the service name from Lambda.
The file system path of an executable binary or script that runs when the Lambda function starts.
AWS_REGION
The AWS Region where the Lambda function is executed.
Adding the Gremlin-Lambda Extension to your Lambda Function
Gremlin releases a new version of the Failure Flags Lambda layer with each sidecar release. This means the Lambda layer version (indicated by the number at the end of the ARN) will change with each release. We’ll include the corresponding Lambda layer number for each sidecar release in the Failure Flags sidecar release notes.
Once you have the correct ARN, add the extension to your Lambda Function by following the instructions in the AWS Lambda Developer Guide.
Lambda extensions can impact the size and performance of your function(s). Gremlin-Lambda's impact is minimal, but non-zero. See the AWS documentation on Lambda extensions for details.
Building your own Lambda Layer
You may want to build your own Lambda Layer or incorporate the gremlin-lambda binary in a Lambda container image. Maybe you're at risk of using too many layers, or maybe your organization has a package caching policy. Either way you'll want to build your own asset. You can access all of the gremlin-lambda and failure-flags-sidecar builds via our website.
Files provided via assets.gremlin.com have the following format: https://assets.gremlin.com/packages/gremlin-lambda/{VERSION}/{ARCH}/{FILE}
You can reference the LATEST release at: https://assets.gremlin.com/packages/gremlin-lambda/latest/{ARCH}/{FILE}
ARCH is one of the following:
x86_64
arm64
Different releases may include different files, but the current FILE set includes:
This Lambda Extension provides debug logging when the GREMLIN_DEBUG environment variable is set to true . This information will be included with your Lambda logs and prefixed with [gremlin-lambda]. This additional information will include configuration details, registration status, connection tracing, any relevant errors encountered while interacting with the control plane, and requests for experiments from the Failure Flags SDKs.
Privileges required
Privilege
Description
CLIENTS_READ
Allows reading all client information within the team
CLIENTS_WRITE
Allows editing all client information within the team
