Fault injection primitives for testing Redis resilience.
Provides tools to kill, pause, partition, and degrade Redis nodes
managed by RedisServerWrapper.Server and RedisServerWrapper.Cluster.
Node-level operations
# Kill a node with SIGKILL
Chaos.kill_node(server)
# Freeze a node (SIGSTOP), then resume later with the returned OS pid
{:ok, os_pid} = Chaos.freeze_node(server)
Chaos.resume_node(os_pid)
# Freeze for a specific duration (auto-resumes)
Chaos.pause_node(server, 5_000)
# Pause all client connections for a duration
Chaos.slow_down(server, 2_000)Cluster-level operations
# Kill the master owning a key's slot
{:ok, killed_pid} = Chaos.kill_master(cluster, "mykey")
# Kill the master owning slot 5000
{:ok, killed_pid} = Chaos.kill_master(cluster, 5000)
# Simulate a network partition (returns frozen OS pids for recovery)
nodes = Cluster.nodes(cluster)
{active, frozen} = Enum.split(nodes, 2)
{:ok, frozen_os_pids} = Chaos.partition(cluster, [active, frozen])
# Undo all chaos (SIGCONT everything)
Chaos.recover(frozen_os_pids)
Summary
Functions
Fills a node with key_count dummy keys (1 KB each) under the chaos:fill:* prefix.
Wipes all data from a node with FLUSHALL.
Sends SIGSTOP to freeze a node indefinitely.
Finds and kills (SIGKILL) the master node owning a given slot or key.
Sends SIGKILL to a server's redis-server OS process.
Simulates a network partition by freezing (SIGSTOP) all nodes not in the first group.
Sends SIGSTOP to freeze a node for duration_ms, then automatically sends SIGCONT.
Kills a random node in the cluster. Returns {:ok, server_pid} of the killed node.
Recovers frozen nodes by sending SIGCONT to each OS pid.
Sends SIGCONT to resume a frozen node.
Pauses all client connections for duration_ms using Redis CLIENT PAUSE.
Forces a replica to take over from its master via CLUSTER FAILOVER.
Triggers a background save (BGSAVE) on the node.
Functions
@spec fill_memory(GenServer.server(), non_neg_integer()) :: :ok | {:error, term()}
Fills a node with key_count dummy keys (1 KB each) under the chaos:fill:* prefix.
Useful for testing memory pressure, eviction policies, and OOM behavior.
@spec flushall(GenServer.server()) :: {:ok, String.t()} | {:error, String.t()}
Wipes all data from a node with FLUSHALL.
@spec freeze_node(GenServer.server()) :: {:ok, pos_integer()} | {:error, term()}
Sends SIGSTOP to freeze a node indefinitely.
Returns {:ok, os_pid} with the OS process ID. Pass the OS pid to
resume_node/1 to unfreeze. The OS pid is needed because the Server GenServer
will be blocked while the redis-server process is frozen.
@spec kill_master(GenServer.server(), String.t() | non_neg_integer()) :: {:ok, pid()} | {:error, term()}
Finds and kills (SIGKILL) the master node owning a given slot or key.
Accepts either a slot number (integer) or a key (string). When given a key, the slot is computed via CLUSTER KEYSLOT.
Returns {:ok, server_pid} with the GenServer pid of the killed node,
or {:error, reason} if the master could not be found.
Examples
{:ok, killed} = Chaos.kill_master(cluster, "user:123")
{:ok, killed} = Chaos.kill_master(cluster, 5000)
@spec kill_node(GenServer.server()) :: :ok | {:error, term()}
Sends SIGKILL to a server's redis-server OS process.
The process is killed immediately and cannot be recovered. Managed Server
processes exit after observing the child death.
@spec partition(GenServer.server(), [[pid()]]) :: {:ok, [pos_integer()]} | {:error, term()}
Simulates a network partition by freezing (SIGSTOP) all nodes not in the first group.
groups is a list of lists of Server GenServer pids (as returned by Cluster.nodes/1).
The first group remains active; all other groups are frozen.
Returns {:ok, os_pids} with the OS pids of frozen nodes. Pass these to
recover/1 to resume them.
Example
nodes = Cluster.nodes(cluster)
{active, frozen} = Enum.split(nodes, 2)
{:ok, frozen_os_pids} = Chaos.partition(cluster, [active, frozen])
# Later...
Chaos.recover(frozen_os_pids)
@spec pause_node(GenServer.server(), non_neg_integer()) :: {:ok, pos_integer()} | {:error, term()}
Sends SIGSTOP to freeze a node for duration_ms, then automatically sends SIGCONT.
The node will be completely unresponsive for the duration, simulating a process freeze
or a very long GC pause. Returns {:ok, os_pid} with the OS process ID.
@spec random_kill(GenServer.server()) :: {:ok, pid()} | {:error, term()}
Kills a random node in the cluster. Returns {:ok, server_pid} of the killed node.
Recovers frozen nodes by sending SIGCONT to each OS pid.
Accepts a list of OS pids as returned by freeze_node/1 or partition/2.
Returns a structured error containing every PID that could not be resumed.
@spec resume_node(pos_integer()) :: :ok | {:error, term()}
Sends SIGCONT to resume a frozen node.
Accepts an OS pid (integer) as returned by freeze_node/1 or partition/2.
@spec slow_down(GenServer.server(), non_neg_integer()) :: {:ok, String.t()} | {:error, String.t()}
Pauses all client connections for duration_ms using Redis CLIENT PAUSE.
Unlike freeze_node/1, the server process stays alive and responsive to health
checks via the admin interface, but all client commands are delayed.
@spec trigger_failover( GenServer.server(), keyword() ) :: {:ok, String.t()} | {:error, String.t()}
Forces a replica to take over from its master via CLUSTER FAILOVER.
The replica argument should be a Server GenServer pid for a replica node.
Options
:force- iftrue, sendsCLUSTER FAILOVER FORCEdirectly (default:false)
When force is false, tries CLUSTER FAILOVER first. If the master is down
(Redis returns a "Master is down or failed" error), automatically retries with
CLUSTER FAILOVER FORCE.
@spec trigger_save(GenServer.server()) :: {:ok, String.t()} | {:error, String.t()}
Triggers a background save (BGSAVE) on the node.
Can be used to test behavior during RDB persistence.