Running Failure Flags experiments

This document will walk you through running your first experiment using Failure Flags.

Example: the HTTPHandler service

Throughout this document, we'll demonstrate examples use a simple service called "HTTPHandler". This service takes incoming web requests and returns the time taken to process each request. We'll provide examples for each of the available SDKs.

We've added a Failure Flag named <span class="code-class-custom">http-ingress</span> with two labels: one that tracks the request method, and one that tracks the URL path:

Node.js example

const gremlin = require('@gremlin/failure-flags')

module.exports.handler = async (event) => {
  start = Date.now()

  // If there is an experiment defined for this failure-flag, that is also
  // targeting the HTTP method and or path then this will express the
  // effects it describes.
  await gremlin.invokeFailureFlag({
    name: 'http-ingress',
    labels: {
      method: event.requestContext.http.method,
      path: event.requestContext.http.path,
    },
  })

  return {
    statusCode: 200,
    body: JSON.stringify(
      {
        processingTime: Date.now() - start,
        timestamp: event.requestContext.time,
      },
      null,
      2
    ),
  }
}

‍

Go example

package main

import (
    "fmt"
    "time"

  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"

  gremlin "github.com/gremlin/failure-flags-go"
)

func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    start := time.Now()

  // Add a failure flag
    gremlin.Invoke(gremlin.FailureFlag{
        Name: `http-ingress`, // The name of the failure flag
        Labels: map[string]string{  // Additional metadata we can use for targeting
              `method`: request.HTTPMethod,
              `path`: request.Path,
        }})

    return events.APIGatewayProxyResponse{
        Body:       fmt.Sprintf(`{"processingTime": %v, "timestamp": "%v"}`, time.Since(start), start),
        StatusCode: 200,
    }, nil
}

func main() {
    lambda.Start(handler)
}

‍

Creating a new Failure Flags experiment

To create a new experiment:

1. Open the Gremlin web app and select Failure Flags in the left-hand nav menu.
2. Click Create an Experiment.
3. Enter an experiment name. This can be anything you wish.
4. Under Failure Flag Selector, use the drop-down menu to select the Failure Flag you want to target.
   1. To limit the code executions that the Failure Flag will impact, use the Attributes drop-down. This drop-down lists labels attached to the Failure Flag, such as the HTTP request method and URL. Select the attribute name from the drop-down, then in the Value field, enter the value(s) that the attribute must match in order to be impacted.
   2. You can enter additional attributes by clicking the + icon, or remove an attribute by clicking the recycling bin icon.
5. Under Service Selector, use the drop-down menu to select the service (i.e. the serverless application) you want to target. This menu will only list actively detected services. You can see a list of active services in the Gremlin web app.
   1. To limit which services the Failure Flag will impact, use the Attributes drop-down. This drop-down lists labels detected in the service itself, such as availability zone or region (for cloud services). Select the attribute name from the drop-down, then in the Value field, enter the value(s) that the attribute must match in order to be impacted.
   2. You can enter additional attributes by clicking the + icon, or remove an attribute by clicking the recycling bin icon.
   3. By default, any new instances of the service that Gremlin detects will be automatically included in the experiment. To prevent this, click the Prevent new services from joining at runtime checkbox.
6. Under Effects, use the drop-down menu to select the type of experiment (the “effect”) to run. Then, use the Attributes boxes to specify the parameters for the experiment. For example, selecting a latency experiment shows two attribute fields: one for the amount of latency to add (ms), and one for the amount of jitter to add (jitter).
7. Select the percentage of matching Failure Flags instances to impact using the Impact Probability boxes. For example, if you choose 1%, then only 1% of the total Failure Flag instances matching your selectors will be impacted by the experiment. This does not apply to code executions—the Failure Flag selector name determines that.
8. Specify how long the experiment will run for using Experiment Duration.
9. Click Save to save the experiment, or Save & Run to save and immediately execute the experiment.

‍

Selectors

Selectors are JSON objects consisting of key-object pairs. These objects tell Gremlin which services and Failure Flags to target for an experiment, as well as what effects to apply.

As an example, our HTTPHandler contains the following Node.js code:

const gremlin = require('@gremlin/failure-flags');

module.exports.handler = async (event) => {
  await gremlin.invokeFailureFlag({
    name: 'http-ingress',
    labels: {
      method: event.requestContext.http.method,
      path: event.requestContext.http.path }});
...
};

‍

This means that the the Failure Flag name is <span class="code-class-custom">http-ingress</span>, and the service name is <span class="code-class-custom">HTTPHandler</span>.

Service Attributes

Service attributes let you identify specific instances of a service to run an experiment on. For example, imagine our <span class="code-class-custom">HTTPHandler</span> service runs in AWS Lambda in several different regions. We can use the following service attribute to only impact instances in <span class="code-class-custom">us-west-1</span>:

{ "region": ["us-west-1"] }

‍

Flag Attributes

Flag attributes are selectors for targeting specific executions of the service's code. Our example <span class="code-class-custom">HTTPHandler</span> service has a method label containing the HTTP request <span class="code-class-custom">method</span>. If we only want to impact <span class="code-class-custom">POST</span> requests, we'd add the following flag attribute:

{ "method": ["POST"] }

‍

Experiments and Effects

The Effect parameter is where you define the details of the experiment and the impact it will have on your service. The Effect parameter is a simple JSON map that gets passed to the Failure Flags SDK when a service is targeted by a running experiment.

The SDK currently supports two types of effects: latency and error.

Latency

Latency introduces a constant delay into each invocation of the experiment. Specify <span class="code-class-custom">latency</span> for the key, and the number of milliseconds you want to delay as the value. For example, this effect introduces a 2000 millisecond delay:

{ "latency": 2000 }

‍

Minimum latency with jitter

Alternatively, you can add latency where the amount varies. For example, this effect introduces between 2000 and 2200 milliseconds of latency, where there is a pseudo-random uniform probability of the SDK applying any value within the jitter amount:

{
  "latency": {
    "ms": 2000,
    "jitter": 200
  }
}

‍

Errors

The Error effect throws an error with the provided message. This is useful for triggering specific error-handling methods or simulating errors you might find in production. For example, this effect triggers an error with the message "Failure Flag error triggered":

{ "exception": "Failure Flag error triggered" }

‍

If your service uses custom error types or other error condition metadata, you can add this metadata to the error effect:

{
  "exception": {
    "message": "Failure Flag error triggered",
    "name": "CustomErrorType",
    "someAdditionalProperty": "add important metadata here"
  }
}

‍

Combining Latency and Error effects

You can combine the latency and error effect to cause a delay before throwing an exception. This is useful for recreating conditions like network connection failures, degraded connections, or timeouts.

For example, this effect will cause the Failure Flag to pause for 2 full seconds before throwing an exception with a custom message:

{
  "latency": 2000,
  "exception": "Failure Flag delayed error triggered"
}

‍

Changing service data

‍

Failure Flags are also capable of modifying data, such as variables. This is an advanced effect that requires additional setup using the Failure Flags SDK.

In your service's call to <span class="code-class-custom">invokeFailureFlag</span>, add a new <span class="code-class-custom">dataPrototype</span> property and assign it a variable like a network request or response. You could also pass in an object literal.

let myData = {name: 'HTTPResponse'}; // this is just example data, it could be anything

myData = await failureflags.invokeFailureFlag({
  name: 'flagname',       // the name of your failure flag
  labels: {},             // additional attibutes about this invocation
  dataPrototype: myData); // "myData" is some variable like a request or response. You could also pass in an object literal.

‍

Once the <span class="code-class-custom">dataPrototype</span> property is set, you can add a <span class="code-class-custom">data</span>object to the effect statement. Any properties in the <span class="code-class-custom">data</span> object will be copied into a new object created from the prototype you provided.

{
  "data": {
    "statusCode": 404,
    "statusMessage": "Not Found"
  }
}

‍

While this experiment is active, <span class="code-class-custom">myData</span> will be changed to the following:

{
  "name": "HTTPResponse",
  "statusCode": 404,
  "statusMessage": "Not Found"
}

‍

‍

Customizing an experiment's impact

You can customize the impact of the experiment by adding a <span class="code-class-custom">behavior</span> function. For example, the following snippet writes data about the experiment to the console instead of applying the experiment to your code:

Node.js example

await gremlin.invokeFailureFlag({
  name: 'http-ingress',
  labels: {
    method: event.requestContext.http.method,
    path: event.requestContext.http.path,
  },

  // Log the experiment after it's complete
  behavior: async (experiment) => {
    console.log('handling the experiment', experiment)
  },
})

‍

Go example

gremlin.Invoke(gremlin.FailureFlag{
  Name: `http-ingress`,
  Labels: map[string]string{
    `method`: request.HTTPMethod,
    `path`: request.Path,
  },

  // the following line provides an implementation of the failureflags.Behavior type
  Behavior: func(ff FailureFlag, exps []Experiment) (impacted bool, err error) {
    // write the experiments to standard out
    fmt.Fprintf(os.Stdout, `processing experiments: %v`, exps)
    // continue processing using the default behavior chain
    return failureFlags.DelayedPanicOrError(ff, exps)
  }
})

‍

If you want even more manual control, the SDK can detect whether an experiment is currently active. For example, during an experiment, you might want to prevent making certain API calls, or rollback a transaction. In most cases the Exception effect can help, but you can also create branches in your code. For example:

Node.js example

if (await failureflags.invokeFailureFlag({ name: 'myFailureFlag' })) {
  // If there is a running experiment then run this branch
} else {
  // If there is no experiment, or it had no impact, then run this branch
}

‍

Go example

if active, impacted, err := FailureFlag{Name: `myFailureFlag`}.Invoke(); active && impacted {
  // If there is a running experiment then run this branch
} else {
  // If there is no experiment, or it had no impact, then run this branch
}

‍

Language-specific features

This section is for features unique to specific SDKs.

Go

Panic

The Go SDK offers a unique fault called <span class="code-class-custom">panic</span>. This causes Failure Flags to panic with the provided message. This is useful when validating that either your service handles Go panics correctly, or when assessing the impact to other parts of the system when your code panics:

{ "panic": "this message will be used in an error provided to panic" }

‍

More information and examples are available on the project's GitHub repo.