Enabling face and object detection

Turn on the face and object detector, so detect and anchor=smart-face crops follow the faces and objects in each image. This guide assumes ImagePipe is running in your app (see Getting started with Phoenix) or as image_pipe_server. anchor=smart works without any of this.

The bundled detector finds faces with YuNet and 80 kinds of objects with RT-DETR. Content-aware cropping explains how the results place the crop.

Install the detector

Plug

Add image_vision and its ONNX runtime, ortex, next to image_pipe:

# mix.exs
defp deps do
  [
    {:image_pipe, "~> 0.1.0"},
    {:image_vision, "~> 0.4"},
    {:ortex, "~> 0.1"}
  ]
end

ImagePipe uses the detector as soon as both dependencies are compiled in. The detector option already defaults to it.

image_pipe_server

Use the -vision variant of the image:

docker run --read-only --tmpfs /tmp -p 8080:8080 -v ./config.toml:/etc/image_pipe/config.toml:ro -v ./images:/data/images:ro ghcr.io/hlindset/image_pipe_server:0.1-vision

Load the models

Plug

Each model downloads the first time it is used. RT-DETR, the object model, is about 175 MB, so download it when you build or deploy, rather than each time a new server starts:

mix image_vision.download_models --detect

Models are stored in image_vision's cache directory, by default under the user's cache directory (~/.cache/image_vision on Linux). If you build a release on one machine or user and run it on another, set the same directory for both:

# config/config.exs
config :image_vision, :cache_dir, "/var/lib/image_vision/models"

An ImagePipe instance loads both models when it starts, so the first detection request doesn't wait for them. It also downloads the face model, YuNet (about 340 KB), and any other model that isn't on disk yet. Loading runs in the background and doesn't delay startup. If you only use face detection, set detector_warmup to skip RT-DETR:

# lib/my_app/application.ex
children = [
  {ImagePipe, name: MyApp.Images, sources: [...], detector_warmup: ["face"]},
  MyAppWeb.Endpoint
]

A mount configured in the router without an instance loads nothing at startup. Add the warmup worker to your supervision tree instead:

# lib/my_app/application.ex
children = [
  {ImagePipe.Transform.Detector.Warmup, detector: :default},
  MyAppWeb.Endpoint
]

image_pipe_server

The -vision image has both models built in and loads them when the server starts. It needs no network access for detection.

Require detection

By default, a detect request that can't run detection still succeeds, cropped with attention cropping instead. To fail those requests instead, set detector_required:

Plug

# lib/my_app/application.ex
children = [
  {ImagePipe, name: MyApp.Images, sources: [...], detector_required: true},
  MyAppWeb.Endpoint
]

image_pipe_server

[processing]
detector_required = true

Error responses lists the statuses those requests get. anchor=smart-face always falls back to attention cropping. Missing or failed detection explains when to require detection.

A configuration that requires detection without a working detector fails before it serves anything, with the message detector_required: the detector is not available in this build:

Plug

ImagePipe.config/1, the ImagePipe child specification, and an ImagePipe.Plug mount configured in the router raise ArgumentError.

image_pipe_server

The image without the -vision variant refuses to start and logs the message.

Check that detection runs

Turn on the default request log:

Plug

# lib/my_app/application.ex, in start/2
ImagePipe.Telemetry.attach_default_logger()

image_pipe_server

[telemetry]
log_requests = true

Then request a face crop of a portrait. For a Plug, add the path you forwarded to ImagePipe.Plug, such as /images:

curl -o face.jpg "http://localhost:8080/w=400/h=400/fit=cover/detect=face/src/woman.jpg"

The log shows the face model running and finding the face:

[info] image_pipe transform detect model: ok (1 regions, ImagePipe.Transform.Detector.ImageVision.Face)
[info] image_pipe transform detect: detected

Other results on the detect line:

• no_regions: the model ran and found nothing, so the crop used attention cropping.
• unavailable, logged as a warning: the detector dependencies aren't installed.
• error, logged as a warning: the model failed on this image.
• skipped (no detector configured), logged as a warning: the detector option is nil.

Next steps

• Crop guides lists the detect and anchor syntax and the class names.
• Detection telemetry describes the detection events, including per-model timings.
• Writing a custom detector replaces the bundled detector with your own model or service.