RedisServerWrapper.Chaos (redis_server_wrapper v0.7.2)

Copy Markdown View Source

Fault injection primitives for testing Redis resilience.

Provides tools to kill, pause, partition, and degrade Redis nodes managed by RedisServerWrapper.Server and RedisServerWrapper.Cluster.

Node-level operations

# Kill a node with SIGKILL
Chaos.kill_node(server)

# Freeze a node (SIGSTOP), then resume later with the returned OS pid
{:ok, os_pid} = Chaos.freeze_node(server)
Chaos.resume_node(os_pid)

# Freeze for a specific duration (auto-resumes)
Chaos.pause_node(server, 5_000)

# Pause all client connections for a duration
Chaos.slow_down(server, 2_000)

Cluster-level operations

# Kill the master owning a key's slot
{:ok, killed_pid} = Chaos.kill_master(cluster, "mykey")

# Kill the master owning slot 5000
{:ok, killed_pid} = Chaos.kill_master(cluster, 5000)

# Simulate a network partition (returns frozen OS pids for recovery)
nodes = Cluster.nodes(cluster)
{active, frozen} = Enum.split(nodes, 2)
{:ok, frozen_os_pids} = Chaos.partition(cluster, [active, frozen])

# Undo all chaos (SIGCONT everything)
Chaos.recover(frozen_os_pids)

Summary

Functions

Fills a node with key_count dummy keys (1 KB each) under the chaos:fill:* prefix.

Wipes all data from a node with FLUSHALL.

Sends SIGSTOP to freeze a node indefinitely.

Finds and kills (SIGKILL) the master node owning a given slot or key.

Sends SIGKILL to a server's redis-server OS process.

Simulates a network partition by freezing (SIGSTOP) all nodes not in the first group.

Sends SIGSTOP to freeze a node for duration_ms, then automatically sends SIGCONT.

Kills a random node in the cluster. Returns {:ok, server_pid} of the killed node.

Recovers frozen nodes by sending SIGCONT to each OS pid.

Sends SIGCONT to resume a frozen node.

Pauses all client connections for duration_ms using Redis CLIENT PAUSE.

Forces a replica to take over from its master via CLUSTER FAILOVER.

Triggers a background save (BGSAVE) on the node.

Functions

fill_memory(server, key_count \\ 10000)

@spec fill_memory(GenServer.server(), non_neg_integer()) :: :ok

Fills a node with key_count dummy keys (1 KB each) under the chaos:fill:* prefix.

Useful for testing memory pressure, eviction policies, and OOM behavior.

flushall(server)

@spec flushall(GenServer.server()) :: {:ok, String.t()} | {:error, String.t()}

Wipes all data from a node with FLUSHALL.

freeze_node(server)

@spec freeze_node(GenServer.server()) :: {:ok, non_neg_integer()}

Sends SIGSTOP to freeze a node indefinitely.

Returns {:ok, os_pid} with the OS process ID. Pass the OS pid to resume_node/1 to unfreeze. The OS pid is needed because the Server GenServer will be blocked while the redis-server process is frozen.

kill_master(cluster, key)

@spec kill_master(GenServer.server(), String.t() | non_neg_integer()) ::
  {:ok, pid()} | {:error, term()}

Finds and kills (SIGKILL) the master node owning a given slot or key.

Accepts either a slot number (integer) or a key (string). When given a key, the slot is computed via CLUSTER KEYSLOT.

Returns {:ok, server_pid} with the GenServer pid of the killed node, or {:error, reason} if the master could not be found.

Examples

{:ok, killed} = Chaos.kill_master(cluster, "user:123")
{:ok, killed} = Chaos.kill_master(cluster, 5000)

kill_node(server)

@spec kill_node(GenServer.server()) :: :ok

Sends SIGKILL to a server's redis-server OS process.

The process is killed immediately and cannot be recovered. The GenServer will still be alive but the underlying redis-server will be dead.

partition(cluster, list)

@spec partition(GenServer.server(), [[pid()]]) :: {:ok, [non_neg_integer()]}

Simulates a network partition by freezing (SIGSTOP) all nodes not in the first group.

groups is a list of lists of Server GenServer pids (as returned by Cluster.nodes/1). The first group remains active; all other groups are frozen.

Returns {:ok, os_pids} with the OS pids of frozen nodes. Pass these to recover/1 to resume them.

Example

nodes = Cluster.nodes(cluster)
{active, frozen} = Enum.split(nodes, 2)
{:ok, frozen_os_pids} = Chaos.partition(cluster, [active, frozen])

# Later...
Chaos.recover(frozen_os_pids)

pause_node(server, duration_ms)

@spec pause_node(GenServer.server(), non_neg_integer()) :: {:ok, non_neg_integer()}

Sends SIGSTOP to freeze a node for duration_ms, then automatically sends SIGCONT.

The node will be completely unresponsive for the duration, simulating a process freeze or a very long GC pause. Returns {:ok, os_pid} with the OS process ID.

random_kill(cluster)

@spec random_kill(GenServer.server()) :: {:ok, pid()}

Kills a random node in the cluster. Returns {:ok, server_pid} of the killed node.

recover(os_pids)

@spec recover([non_neg_integer()]) :: :ok

Recovers frozen nodes by sending SIGCONT to each OS pid.

Accepts a list of OS pids as returned by freeze_node/1 or partition/2. Nodes that were killed (not just frozen) are silently skipped.

resume_node(os_pid)

@spec resume_node(non_neg_integer()) :: :ok

Sends SIGCONT to resume a frozen node.

Accepts an OS pid (integer) as returned by freeze_node/1 or partition/2.

slow_down(server, duration_ms)

@spec slow_down(GenServer.server(), non_neg_integer()) ::
  {:ok, String.t()} | {:error, String.t()}

Pauses all client connections for duration_ms using Redis CLIENT PAUSE.

Unlike freeze_node/1, the server process stays alive and responsive to health checks via the admin interface, but all client commands are delayed.

trigger_failover(replica, opts \\ [])

@spec trigger_failover(
  GenServer.server(),
  keyword()
) :: {:ok, String.t()} | {:error, String.t()}

Forces a replica to take over from its master via CLUSTER FAILOVER.

The replica argument should be a Server GenServer pid for a replica node.

Options

  • :force - if true, sends CLUSTER FAILOVER FORCE directly (default: false)

When force is false, tries CLUSTER FAILOVER first. If the master is down (Redis returns a "Master is down or failed" error), automatically retries with CLUSTER FAILOVER FORCE.

trigger_save(server)

@spec trigger_save(GenServer.server()) :: {:ok, String.t()} | {:error, String.t()}

Triggers a background save (BGSAVE) on the node.

Can be used to test behavior during RDB persistence.