Face-aware crops, zoom, and pixelation

This guide describes how image_plug integrates with the optional image_vision library to add face-aware behaviour to the IR. It covers the seam (Image.Plug.FaceAware), the per-provider URL grammar that drives it (gravity=face, face-zoom=…, z_…, z-…, crop=faces), the runtime model loading, and the graceful fallback when image_vision is not in the consumer's dependency tree.

Why a seam, not a hard dep

image_plug does not declare :image_vision as a dependency. The face-detection model — YuNet ONNX, ~340 KB on disk plus the Ortex / Nx / EXLA runtime — adds tens of megabytes to the dependency closure and ML build complexity that not every consumer wants. Apps that don't need face-aware behaviour pay nothing.

The integration point is Image.Plug.FaceAware. It calls into Image.FaceDetection only after a runtime Code.ensure_loaded?/1 check, with @compile {:no_warn_undefined, Image.FaceDetection} so the build stays clean when the optional dep is absent. When image_vision is not loaded, every function in the seam returns {:error, :unavailable} and the interpreter routes around it transparently — face-aware requests fall back to libvips' attention-based saliency crop, which is a reasonable approximation in most cases.

What gets face-aware behaviour

Two operations in the IR consume face detections:

• Ops.Resize{gravity: :face, face_zoom: float} — pre-crops the image to the most prominent detected face plus padding (derived from face_zoom) before the regular thumbnail resize pass. The thumbnail then sees a face-centred region and produces a face-centred result. With :image_vision absent or with no detected face, the existing :attention saliency crop takes over.

• Ops.PixelateFaces{scale: float} — detects every face above the confidence threshold and pixelates each region in place, leaving the rest of the image untouched. With :image_vision absent, the op is a no-op (the original image is returned unchanged).

URL grammar across the four providers

The same IR field — Resize.face_zoom, a float in [0.0, 1.0] — is expressed differently in each provider's URL:

Gravity itself rides alongside:

Image.Components.URL.<provider>/2 emits the right token for each. The four URLs round-trip back to the same IR through the matching parser in image_plug.

Semantics of face_zoom

face_zoom is the tightness of the face-aware crop, not a literal zoom factor:

• face_zoom = 0.0 (the default) gives a loose crop — the bounding box is expanded by a full padding = 1.0 on each side, which often means the entire image. The downstream resize pass then sees the same input as a normal gravity=center request, and the output is indistinguishable from a centred crop. This is the most common reason a user reports "I picked gravity=face and nothing happened" — the answer is to also give face_zoom a non-zero value.

• face_zoom = 0.6 (Cloudflare's documented default) gives a moderately tight crop with some context around the face. This is a good general-purpose setting for portrait thumbnails.

• face_zoom = 1.0 hugs the face bounding box — the resulting crop is essentially the face with no surrounding context.

The mapping inside Image.Plug.FaceAware.face_crop/2 is padding = max(1.0 - face_zoom, 0.0), so the relationship between face_zoom and bounding-box padding is linear and inverse.

Behaviour when no face is detected

Image.FaceDetection.crop_largest/2 returns {:error, :no_face_detected} when nothing scores above the YuNet confidence threshold (default 0.6). The interpreter catches this and falls through to the normal thumbnail flow with the original image and the original gravity setting. The request still succeeds — you don't see a placeholder; you get a saliency-based crop.

Model loading

The first request that invokes face detection downloads the YuNet ONNX weights (face_detection_yunet_2023mar.onnx, ~340 KB) from HuggingFace into image_vision's model cache directory:

• On macOS: ~/Library/Caches/image_vision/opencv/face_detection_yunet/
• In a container: configure via :image_vision, :cache_dir (typically a mounted volume — see the image_playground Dockerfile).

The download happens once per cache directory. Subsequent requests load the model from :persistent_term and run inference on every face-aware request.

Wiring it into your app

Add :image_vision to your dependencies:

def deps do
  [
    {:image_plug, "~> 0.1"},
    {:image_vision, "~> 0.3"},
    # image_vision's ML stack:
    {:ortex, "~> 0.1"},
    {:nx, "~> 0.10"},
    {:exla, "~> 0.10"}
  ]
end

That's it — no further configuration is required. Image.Plug.FaceAware.available?/0 flips to true on next boot and the interpreter starts honouring gravity: :face and Ops.PixelateFaces. To verify end-to-end, point a Cloudflare URL at a known face image:

GET /cdn-cgi/image/width=300,height=300,fit=cover,gravity=face,face-zoom=0.6/portrait.jpg

Compare the bytes against the same URL with gravity=center,face-zoom=0 (the default) on an image that has a face well off-centre — different MD5s prove face detection ran.

Related

• Image.Plug.FaceAware — the seam itself; available?/0, face_crop/2, pixelate_faces/2.
• Image.FaceDetection — the underlying detector in image_vision.
• image_components/guides/usage.md — how to drive face_zoom through the <.image> Phoenix.Component.
• image_playground — exercises this end-to-end with a slider per parameter.