Container security

Gremlin containers run as root

When Gremlin runs within a container, <span class="code-class-custom">gremlin</span> processes run as <span class="code-class-custom">root</span>. This is because there are several file resources that Gremlin must mount from the host which are owned by <span class="code-class-custom">root</span> on the host machine:

• <span class="code-class-custom">/var/lib/gremlin</span> and <span class="code-class-custom">/var/log/gremlin</span>: These are mounted from the host into the Gremlin agent container and sidecar experiments. By default, container runtimes will set the ownership of these resources to <span class="code-class-custom">root:root</span>Container runtime sockets (e.g. <span class="code-class-custom">/var/run/docker.sock</span>), which by default will be owned by <span class="code-class-custom">root</span>

• <span class="code-class-custom">/proc/sysrq-trigger</span> and <span class="code-class-custom">/sys/fs/cgroup</span> for attack capabilities, which by default will be owned by root

Mitigating the privileges of root

Though Gremlin runs as <span class="code-class-custom">root</span> within the container in which it runs, Gremlin is restricted by the specific capabilities it requests, despite running as <span class="code-class-custom">root</span>. Here are some other mitigations system administrators can make to reduce the impact of Gremlin's usage of <span class="code-class-custom">root</span>:

• While Docker's default seccomp profile is already fairly restrictive, you can remove any undesired syscalls by overriding this default, or provide a more restrictive seccomp profile to Gremlin containers.
• Restrict Gremlin's resource access on the host using AppArmor and the Gremlin Agent AppArmor Profile.
• Consider an extra layer of safety by enabling SELinux to harden all containers running on your systems

Alternatives

If you do not wish to run Gremlin as a <span class="code-class-custom">root</span> user, even with the restrictions described above, consider installing Gremlin directly onto the host where it runs under a dedicated Linux user: <span class="code-class-custom">gremlin</span> by default. Note however that this will require changes to the host's file system to allow access to container runtime sockets such as <span class="code-class-custom">/var/run/docker.sock</span> and will generally produce an outcome similar to running Gremlin in a container.

Docker (Linux)

User namespace isolation

Gremlin currently uses the host's file system to store temporary log and state information about experiments. When running Docker with user namespace remapping (<span class="code-class-custom">userns-remap</span>), Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running <span class="code-class-custom">gremlin experiment-container</span>. Note that by assuming the user namespace of the host, we are creating an exception to backspace isolation for the Docker containers running Gremlin.

For running the Gremlin daemon in a container

docker run -d \
    --userns-remap=host \
    -e GREMLIN_BYPASS_USERNS_REMAP=1 \
    -v /var/lib/gremlin:/var/lib/gremlin \
    -v /var/log/gremlin:/var/log/gremlin \
    gremlin/gremlin daemon

‍

For running the Gremlin daemon on the host

echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind
sudo systemctl restart gremlind

‍

For running a Gremlin experiment from the command line

export GREMLIN_BYPASS_USERNS_REMAP=1
gremlin attack-container 38dbd9016529 cpu

‍

SELinux and Gremlin in Containers

Gremlin performs some actions that are not allowed by the default SELinux process label for containers (<span class="code-class-custom">container_t</span>):

• Install and manipulate files on the host: <span class="code-class-custom">/var/lib/gremlin</span>, <span class="code-class-custom">/var/log/gremlin</span>
• Load kernel modules for manipulating network transactions during network experiments, such as <span class="code-class-custom">net_sch</span>
• Communicate with the container runtime socket (e.g. <span class="code-class-custotm">/var/run/docker.sock</span>) to launch containers that carry out experiments
• Read files in <span class="code-class-custom">/proc</span>

Bypass container_t restrictions

It is possible to alleviate these restrictions on <span class="code-class-custom">container_t</span> by installing the following policy. However, this grants the privileges required by Gremlin to all other containers on your system that use <span class="code-class-custom">container_t</span>.

‍

If you wish to run Gremlin with the <span class="code-class-custom">container_t</span> process label, and bypass its restrictions, supply the following type enforcement rules into a new SELinux policy:

# WARNING: This policy adds capabilities to all containers run under the default type: container_t
# Gremlin needs access to /var/log/gremlin
allow container_t container_log_t:dir { read write create getattr setattr unlink link add_name remove_name rmdir open };
allow container_t container_log_t:file { read write create getattr setattr append unlink link open };
allow container_t var_log_t:dir { write add_name };

# Gremlin needs access to /var/run/docker.sock
allow container_t container_runtime_t:unix_stream_socket connectto;

# Gremlin needs access to /var/lib/gremlin
allow container_t container_var_lib_t:dir { read write create getattr setattr unlink link add_name remove_name rmdir open };
allow container_t container_var_lib_t:file { read write create getattr setattr append unlink link open };

# Gremlin needs to load the kernel modules: net_sch
allow container_t kernel_t:system module_request;

‍