ImagePipe.Cache.FileSystem (image_pipe v0.1.0)

Copy Markdown View Source

Cache adapter that stores processed images, or originals, as files on local disk.

cache: {ImagePipe.Cache.FileSystem, root: "/var/cache/image_pipe/processed"}

Without :max_size_bytes the cache grows without limit. With it, the cache runs in bounded mode, with :max_size_bytes as a soft cap: writes evict the least valuable entries, and a background pass evicts any overshoot. Entries requested most often are kept longest (a W-TinyLFU policy). Bounded mode runs processes that track the cache's size, so a bounded cache must be configured on an instance, which starts them (see ImagePipe.child_spec/1):

children = [
  {ImagePipe,
   name: MyApp.Images,
   sources: sources,
   cache:
     {ImagePipe.Cache.FileSystem,
      root: "/var/cache/image_pipe/processed",
      max_size_bytes: 5_000_000_000,
      node_id: "node-0"}},
  MyAppWeb.Endpoint
]

To set up processed-image and originals caches, see Caching processed images.

Options

  • :root (absolute path (String.t/0)) - Required. Directory that holds the cache files. Must be an absolute path. The cache and input_cache roots must be separate, neither inside the other. Plug initialization fails otherwise.

  • :path_prefix (relative path (String.t/0)) - Subdirectory of :root where entries are written. Must be a relative path without backslashes, . or .. segments, or empty segments (such as a//b or a trailing /). The default value is "".

  • :pool (:input or :output) - Tags bounded-mode telemetry events with pool, so events from an originals cache can be told apart from the processed-image cache. Defaults to :output. A cache configured as :input_cache gets pool: :input automatically.

  • :max_size_bytes (pos_integer/0) - Soft cap, in bytes, on the total size of stored responses. Setting it turns on bounded mode, which requires :node_id. Every option below requires :max_size_bytes.

  • :node_id (String.t/0) - Name of this node, used for its state file <node_id>.state. Required in bounded mode. Must stay the same across restarts and differ between nodes that share a root.

  • :state_dir (String.t/0) - Directory for the per-node state files. On boot a node merges the request counts from every peer state file younger than :state_ttl. Defaults to <root>/.cache_state.

  • :window_ratio (float/0 from 0.0 to 1.0) - Share of :max_size_bytes reserved for newly stored responses. A larger ratio favors recently added responses over often-requested ones. 0.0 reserves no space. Defaults to 0.01.

  • :sketch_depth (pos_integer/0) - Number of counter rows used to count requests per response. More rows count more accurately but use more memory. Defaults to 4.

  • :sketch_width (pos_integer/0) - Number of counters per row. More counters count more accurately but use more memory. Defaults to one per 25,000 bytes of :max_size_bytes, and at least 4,096.

  • :aging_sample_size (pos_integer/0) - Number of counted requests after which every count is halved, so old popularity fades. A smaller value forgets faster. Defaults to one per 5,000 bytes of :max_size_bytes, and at least 81,920.

  • :doorkeeper_cardinality (pos_integer/0) - Number of distinct responses the node tracks as requested once. A response is counted only from its second request, so one-off requests don't push out popular responses. Defaults to one per 12,500 bytes of :max_size_bytes, and at least 8,192.

  • :doorkeeper_fpr (float/0 greater than 0.0 and less than 1.0) - Rate at which a first request is mistaken for a repeat and counted early. Lower rates use more memory. Defaults to 0.01.

  • :eviction_victim_limit (pos_integer/0) - Maximum number of entries one write or one background eviction batch removes. A write that needs to evict more removes this many and starts background eviction for the rest, so the cache can briefly exceed :max_size_bytes. If the entries it needs to evict include ones read recently, the write is rejected instead. Defaults to 64.

  • :flush_interval (pos_integer/0) - Seconds between writes of this node's state file. Defaults to 30.

  • :cleanup_interval (pos_integer/0) - Seconds between removals of stale peer state files. Defaults to 3600.

  • :reconcile_interval (pos_integer/0) - Seconds between background passes that evict entries until the stored size is at or under :max_size_bytes. Defaults to 60.

  • :rescan_interval (pos_integer/0) - Seconds between background passes that list the cache directory, so this node counts entries other nodes sharing :root wrote or removed, and then delete left-over files. Each pass is delayed or advanced by up to a fifth of the interval at random. Defaults to 300.

  • :state_ttl (pos_integer/0) - Seconds after its last change before a peer state file counts as stale. Stale files are ignored on boot and removed during cleanup. Defaults to 604_800 (seven days).

The adapter also accepts the shared :max_body_bytes option, a non-negative integer or nil. Responses larger than :max_body_bytes are delivered but not stored. The default nil stores responses of any size.

Summary

Functions

Reads a cached response and materializes its complete body as a binary.

Functions

get(key, opts)

Reads a cached response and materializes its complete body as a binary.

Returns {:hit, entry}, :miss, or {:error, reason}. The cached file's size is checked before reading, and its descriptor is closed before returning.