- New Introducing experiment GPU, which will attempt to exhaust all compute resources for the system's GPUs.
- Info Updated dependencies
- New Dependency Discovery now uses open connections in combination with DNS records to discover only hostnames for which the Gremlin Agent has seen an active connection. Users should no longer see dependencies discovered for services that only make DNS queries to a particular hostname, but never actually connect to them.
- Info Updated dependencies
- Info Improved logging around actions the Gremlin agent takes to rollback active experiments.
- New Blackhole experiments now automatically add the route to the Gremlin service to the exclusion rules so that connection is maintained during the attack. This can be disabled if needed via
--no-derived-exclusion-rules. - Info Updated dependencies
-
Fix Fixed bugs with the Gremlin Agent's session renewal process, cleaning up spurrious
Unauthorizederrors and preventing rare instances where local session storage can get corrupted. - Info Updated dependencies
- New The Gremlin Agent now reports the DNS servers that are used by the host on which it is installed. This data is used to select a random DNS server to impact for our new
Redundancy: DNSreliability test, available from the Well-Architected Cloud Test Suite. - Fix Upon receiving a shutdown signal, the Gremlin Agent will wait for any running attacks to finish halting before shutting down. This fixes an issue where attacks would end up
FailedorLostCommunicationinstead ofClientAbortedwhen the Gremlin Agent was terminated during such attacks. - Fix A more detailed error message is provided at install time when any MSI actions fail.
- Info Updated dependencies
-
Fix Improved logging in situations where
gremlindcannot open or parse configuration and certificate files. - Info Updated dependencies
- InfoAdd logging for troubleshooting Windows validation errors.
- InfoBetter error message for failing to load certificates.
- InfoUpdated dependencies.
- NewWindows support for experiment Process Exhaustion, a way to consume processes to identify limits within the target system.
- NewAdded DNS dependency feature to Windows installer. It will be enabled automatically if a valid Windows Pcap library is already installed.
- FixRolling back network attacks no longer considers missing network devices as a critical error. This accounts for failure modes where the network device is torn down externally.
- NewDuring a rollback, the
gremlindprocess tells the associated attack told to tear down before it attempts to remove any remaining impact. - InfoImproved logging in
daemon.logwhen attacks are rolled back. - InfoRaised the TCP connect timeout for API requests that transition attacks between stages from 1 second to 5 seconds.
- FixAddressed an issue where rollback would fail when no teardown was required.
- FixInstall the right Windows driver that matches the KMDF (Kernel-Mode Driver Framework) version.
- NewAdded a new DNS-based dependency collection feature. Learn more about this feature here.
- FixPrint full error on rollback failures.
- InfoUpdated dependencies.
- Fix: Resolved latency issues on ESXi virtual machines
- Info: Updated dependencies
- New Provided Gremlin has access to a valid AWS credentials chain, it now interprets AWS ARN values in
GREMLIN_TEAM_ID,GREMLIN_TEAM_SECRET,GREMLIN_TEAM_CERTIFICATE_OR_FILE,GREMLIN_TEAM_PRIVATE_KEY_OR_FILE. Gremlin supports ARN values from AWS Secrets Manager or AWS Systems Manager Parameter Store. Gremlin can optionally be supplied withGREMLIN_IAM_ROLEto specify a role to assume for the strict purpose of fetching secret values. - Fix More context is added to various error messages
- Fix: Regression introduced in 1.11.0 where attacks with invalid arguments would end up
Lost Communicationinstead ofFailed - Info Updated dependencies
- New Attacks can sometimes fail to notify the Gremlin Control Plane when its connection is impacted by the attack itself. The Gremlin agent now tolerates these failures more often and attempts to resend failed notifications. This fixes attacks that end up in the
HaltFaledstage that would otherwise finish in theSuccessfulstage.
- New Improved the output of the Gremlin Agent validation routine that happens on startup. When validation fails, details about the failure are written to
daemon.log.
- Fix Fixed an issue where attacks were incorrectly labeled
HaltFailedwhen Gremlin fails to notifyapi.gremlin.comduring teardown of the network impact. - New For users running Gremlin on AWS, more error information is printed to the log file when AWS metadata cannot be retrieved.
- Info Updated dependencies
- Fix Errors related to spawning subprocesses now have more detailed information useful for troubleshooting.
- Fix For hostnames supplied to network attacks, Gremlin delegates DNS queries to the operating system. When this query fails, Gremlin now attemps to resolve the name completely within the running process in an attempt to overcome operating system failures. This allows Gremlin network attacks to continue in the face of failed DNS processing.
- Fix Fixed a bug where Latency attacks would attempt to target multiple network interfaces, which is not yet supported. An error is now returned if
--deviceis used to specify multiple network interfaces for a Latency attack. If no--deviceis specified, Gremlin will choose the highest priority device it finds. - Info Updated dependencies.
- Fix Gremlin now tears down the TCP connection pool with
api.gremlin.comafter successive timeout failures. - Fix Gremlin includes the name of the targeted network interface in execution log events related to applying network impact.
- Info Updated dependencies.
- Info Added support for tag values to be any simple YAML datatype (boolean, integer, float, string). Previously only strings were supported.
- Info Updated dependencies.
- Fix Fixed an issue that prevented Gremlin from ingesting Azure Tags.
- Fix Fixed an issue that made Gremlin validation unreliable.
- Info Updated dependencies.
- New Gremlin's version command now prints more build information.
- Info Updated dependencies.
- New Multiple network interface attacks are now supported. Details are available in Network device selection.
- Info Updated dependencies.
- Fix Fix a bug in collect_certs when the target dropped the network connection before completing the TLS setup.
- Fix Only consider IPv4 addresses for the default Gremlin Identifier.
- Info Updated help URLs.
- Info Updated dependencies.
- Fix Fix a bug that prevented collect_certs from working when run against a container.
- Info Updated dependencies.
- New Add a short argument (-n) for the not_less_than option.
- Info Updated dependencies.
- New Introduce Certificate Expiry test for Reliability Management.
- Info Updated dependencies.
- Fix Fixed a bug where Gremlin would not properly launch attacks that resolve to a large amount of IP addresses / blocks.
- Fix: Fixed a bug where Gremlin would not properly include swap in free memory calculations, leading to incorrect attack results.
- Info: Updated dependencies.
- Fix: Fixed a bug where Gremlin would attempt to allocate more memory than was available when the cgroup attribute
memory.limit_in_byteswas higher than available system memory. - Info: Updated dependencies.
- New: Gremlin's Memory attack now has a new argument:
--allocation-strategy(-s), which informs Gremlin on how to interpret other memory consumption arguments:--percent,--mb, and--gb. See more at Memory: Options - Info: Updated dependencies.
- Fix: Correctly handle proxy usernames and passwords that contain special characters. Special characters must be percent-escaped. For example, %5C is used in place of a backslash. Details are available here.
- Info: Updated dependencies.
- New: Disk attack improvements. The Disk attack is much faster, more accurate, and safer.
- New: Gremlin now reads custom Azure tags associated with the machine and makes those tags available for targeting.
- Fix: Fixed a bug where Gremlin would fail validation if the DNS lookup of api.gremlin.com failed. This was likely to be a problem in high security environments.
- Fix: Fixed a bug where Gremlin would crash on some operating systems when process collection was enabled. Gremlin avoids crashing and disables process collection when errors are detected.
- Fix: Fixed a bug where Gremlin used to skip tag configuration even when it had a valid session. Gremlin now always configures tags on startup as long as it can communicate with the Gremlin control plane.
- Fix: Some automatic Azure tags were not being correctly read. The azEnvironment, location, name, osType, privateIpAddress, publicIpAddress, sku, vmId, vmSetScaleName, and zone tags are all automatically read and available for targeting.
- Info: Removed NTP timestamp from
gremlin check os. - Info: Updated dependencies.
- New: On startup, the Gremlin agent now performs some validation on its ability to run a CPU and Latency attack. Validation results are accessible through the Clients API.
- Info: Updated dependencies
- New: The Disk attack has been significantly improved. In most cases it is much faster, more accurate, and safer. It also uses significantly less CPU and RAM when filling disk volumes. The improved version is used when the environment variable
GREMLIN_DEUCHAINN_EN1023is set totrue; all other values are treated asfalse. This environment variable may be ignored or removed in a future version without notice.
- Fix: Fixed bug with Gremlin's IO attack cleanup when
--mode ror--mode wwas used. Previously, Gremlin would try to tear down files that did not exist, leading to attack failures. - Info: Improve messages reported by the Gremlin IO attack, when file-creation errors occur.
- Info: Changed the way
PUSH_METRICSboolean configuration variable is evaluated. Previously, any non-empty value other than"0"would evaluate totrue(e.g.PUSH_METRICS=falsewould evaluate totrue). This has been changed to provide expected outcomes: the only values that evaluatetrueare now"1","true", and"TRUE", leaving all other values to evaluate tofalse. - Info: Updated dependencies
- Fix: Fixed a bug where the Gremlin agent does not properly roll back time travel attacks with an offset of 5 seconds or less.
- New: The
percentargument for Disk attacks now accepts real numbers. For example,--percent 27.5was previously unsupported. - Fix: Gremlin now correctly determines the local hostname making the automatic
local-hostnameavailable for targetting. - Fix: API interactions made by the Gremlin agent now always send the appropriate
Content-Typeheader value. - Info: Updated dependencies
- Fix: Fixed a bug where the CPU attack would not affect all processors on systems with more than 64 processors.
- Info: Updated dependencies
- Fix: This update fixes Memory attack bugs. Previously, the amount of memory consumed would be limited to RAM. Memory attacks with this update include swap space / all virtual memory.
- New: The Memory attack is more "aggressive" in the sense that the memory allocated by Gremlin during the attack is more difficult to swap to disk.
- New: The Gremlin CLI now has a
gremlin check daemonsubcommand which reports on the status of any running Gremlin agent. - Info: Updated dependencies.
- Fix While never observed, according to the Windows API documentation, getting the current username can fail. If that happened the Gremlin Client would fail to run an attack. Instead, this version resorts to using "unknown" if the username cannot be determined.
- New Daemon log file management improvements. Previously, the log file was truncated at midnight. That made troubleshooting difficult. The log file is now rolled when it reaches approximately 1 MiB. Ten compressed log files are kept. With this update the current log file typically captures several days and the compressed log files typically capture a few weeks at a modest cost of approximately 2 MiB of disk space.
- Fix Fix a bug in Gremlin's argument parsing for the
hostnamesandipaddressesarguments for network attacks.
- Fix Improve command-line argument parsing by providing better error messages and catching more edge cases related to illegal inputs.
- Fix Patch a vulnerability in a 3rd party library that posed a variety of memory corruption scenarios, most likely use-after-free.
- Info Improve error messages among network attacks when an invalid network device is supplied. Error message now includes all valid devices.
- Info Drop invalid targeting tags with a warning.
- New The daemon version is included in the
gremlin checkreport. - Fix Occasionally the Docker version was incorrectly parsed which would result in the classic driver being used for container attacks.
- New Some agent API traffic is now gzip-compressed, reducing network overhead on machines where Gremlin is installed.
- Fix Patch a vulnerability in a 3rd party library that posed a potential buffer overflow scenario
- Fix Patch a vulnerability in a 3rd party library that posed a potential scenario to operate on dangling memory references
- New You can now specify the
SSL_CERT_FILEvariable via theconfig.ymlfile. See the advanced configuration page for details on how to use it.
- Fix Gremlin now properly interprets escaped newline characters
\nfor values of theGREMLIN_SSL_CERTenvironment variable. - Info Gremlin now reports container and process data at a slower rate, down from every 5 seconds during active attacks (and every 10 seconds otherwise) to every 30 seconds. We've found that this data changes much less frequently than is justified for a 5-10 second interval. This should result in significantly reduced network overhead required to run Gremlin.
- Info Updated dependencies
- Fix The Gremlin agent now writes a message to
daemon.logwhen attacks finish. This provides observers of this log with an approximation on when attacks have ended. - Info Updated dependencies