Test Suites

A Test Suite is a group of reliability tests that get applied to each service in a Gremlin team. Test Suites let you customize how each of your teams evaluates their reliability scores. For example, you can assign unique Test Suites to one or more teams, or use a single Test Suite across your entire organization.

Test Suites are built on Scenarios. Each Scenario appears in the suite as a single test. You can also group Scenarios into categories, such as "Redundancy", "Scalability", and "Depdendencies." Gremlin also provides a default Test Suite called Gremlin Recommended Tests, which uses our built-in Scenarios to provide a balanced collection of test.

‍

Built-in Test Suites

These Test Suites were created by Gremlin and are available to all Gremlin customers. For details on the reliability tests in each suite, see Reliability Tests.

‍

Gremlin Recommended Tests

This is the default Test Suite for new Gremlin Teams. This Test Suite provides a balanced collection of tests based on reliability best practices. Click here to view this Test Suite in the Gremlin web app.

‍

Well-Architected Cloud Test Suite

Gremlin’s recommended test suite for all cloud-based services. This Suite lets you test and govern to cloud reliability principles and best practices. It includes the tests in Gremlin’s Recommended Test Suite, plus additional tests that cover cloud-specific reliability risks. Click here to view this Test Suite in the Gremlin web app.

‍

‍

Creating and editing a Test Suite

Before creating a new Test Suite, you'll first need to create the Scenarios that you want to add to the Test Suite. Follow the instructions in Creating a Scenario if you haven't yet done that.

Once you have your Scenarios created, follow these steps:

1. Open the Test Suites page in the Gremlin web app.
2. Select + Test Suite.
3. ^Alternatively, you can click Clone next to an existing suite to create a copy. If you don't want to create an entire suite from scratch, cloning the Gremlin Recommended Tests is a great way to get started!
4. Enter a Name for the Test Suite.
5. Under Test Suite Composition, use the Search Scenarios box to find the Scenario you want to add to the suite, then click + Add. The Scenario will appear in the list below the search box.
6. ^You can remove a Scenario from the suite by clicking the Delete button. This won't delete the Scenario; it will simply remove it from the list.
7. ^Optionally, click the Edit button to change how the Scenario appears in the suite. You can change its name, add a description, and choose a category for the Scenario. Note that changing these settings won't change the Scenario itself, only how the Scenario appears in the Test Suite. You can safely make changes here without affecting the base Scenario.
8. ^Click Save to save any edits you made to the Scenario.
9. Under Teams, use the Search Teams box to find the Gremlin team that you want to assign this Test Suite to. You can add multiple teams at once.
10. ^Note: each team can only have one Test Suite. Changing a team's Test Suite will also reset the test results and scores to zero, however, the data for the previous Test Suite is preserved and will reappear if you change back to the original Test Suite later.
11. ^You can remove a team by clicking Remove next to the team name.
12. Check the Test Preview section to confirm that this is the correct Scenario, then click Save to save the Test Suite.

‍

Deleting a Test Suite

‍

To delete a Test Suite, open the Test Suites page, find the suite you want to delete, then click the Delete button. Confirm, and the Test Suite will be deleted. This action will not modify any of the Scenarios that were part of the Test Suite.

‍

Including/excluding Detected Risks in Test Suites

You can include and/or exclude individual Detected Risks in your custom Test Suites. Including a Detected Risk in a Test Suite factors that risk into the reliability score calculations for that Test Suite. By default, all current Detected Risks and any newly-added risks will be included in Test Suites automatically.

When a Test Suite has risks associated with it, each service’s Reliability Score card will show a new Risks category. Each service will also show the number of unmitigated risks. You can click on the Detected Risks card to see the full list of Detected Risks included in this Test Suite, and whether it’s Mitigated or At Risk for this service.

To include or exclude Detected Risks in your Test Suite:

1. In the left-hand navigation pane, open Test Suites, or click this link.
2. Select the Test Suite you wish to edit.
3. Select the Risks tab. This displays two lists: the left list contains the risks included in the Test Suite, while the right list contains the risks that aren’t included. The risks on the left will be tested against each of your services and factored into their reliability score.
   1. From the right-hand column, select the risk(s) you want to include by clicking in the checkbox next to its name.
   2. Click on the < button to move your selected risk(s) to the “Include” column. Alternatively, click the << button to move all risks.
   3. Conversely, use the > or >> buttons to move risks from the “Include” column to the “Ignore” column.
4. Click Save to save your selection.

The next time you access a service’s overview page, you’ll see the new set of Detected Risks listed.