ImagePipe.Transform.Detector behaviour (image_pipe v0.1.0)

Behaviour for detectors that find faces and objects for detect and anchor=smart-face crops.

The default detector, ImagePipe.Transform.Detector.Composite, sends the face class to ImagePipe.Transform.Detector.ImageVision.Face and object classes to ImagePipe.Transform.Detector.ImageVision.Objects. To use another model or a remote service, implement this behaviour and set the :detector option of ImagePipe.config/1 to your module:

config = ImagePipe.config(sources: sources, detector: MyApp.Detector)

Writing a custom detector walks through an implementation. Content-aware cropping explains how the regions place the crop.

Callback options

Every callback receives a keyword list. Its :classes key holds the requested class names, in their URL spelling ("face", "traffic_light"), or :all. Other keys may be present. A detector ignores the keys it doesn't use. supported_classes/1 is called with [].

Regions

detect/2 returns a list of region/0 maps.

label must be one of the requested classes, or any class when :classes is :all. ImagePipe doesn't filter the regions, so a region with another label still moves the crop. Request weights such as face:3 apply by label.
box must hold four numbers. Any other shape makes the whole result a detection error.
A box with no width or height, or one that extends past the image's edges, is ignored.
score doesn't affect the crop, so a low-confidence region moves it as much as a confident one.

Errors

detect/2 returns {:error, reason} instead of raising. With detector_required: true, any error fails the request with 500. With detector_required: false, the crop falls back to attention cropping, and the response depends on reason:

{:detector, :unavailable} means the detector can't run in this build, for example because an optional dependency is missing. The response is cached like any other. identity/1 must return a different value while the detector is unavailable, or the fallback is stored under the identity of the working model.
Any other reason is a detection error. The response is sent with Cache-Control: no-store and no ETag, and isn't stored.

A detector that runs several models returns an error when any of them fails, not the regions the others found. A partial result is cached as if it were complete.

A detector may instead let an unavailable model contribute no regions, as ImagePipe.Transform.Detector.Composite does. Its identity/1 must then change while the model is missing, so the result is cached apart from the complete one.

anchor=smart-face falls back to attention cropping on any error, whatever :detector_required is.

Availability and readiness

available?/1 and the optional ready?/1 are checked before a request fetches its source, only when :detector_required is true and the request uses detect:

available?/1 returning false fails the request with 501.
ready?/1 returning false fails it with 503. Without ready?/1, the detector is ready whenever it is available.

ready?/1 should return true once warmup/1 has succeeded. An ImagePipe instance calls warmup/1 at startup unless its detector_warmup option is false.

With detector_required: true, ImagePipe.config/1 also calls available?/1 with classes: [class] for each supported class, and raises ArgumentError when it returns false for all of them.

With detector_required: false, requests call detect/2 directly. ImagePipe.Transform.Detector.Warmup also skips a detector whose available?/1 returns false.

Identity

identity/1 returns {module, term}, which becomes part of the cache key and ETag of every response that used the detector. Change the term when the model changes, for example by including the model version, so crops made by the old model are replaced. It must return the same value for the same model and options. Keep it free of secrets: telemetry handlers can receive it.

Summary

Types

region()

A detected region of interest.

Callbacks

available?(opts)

Returns whether the detector can run in this build, for example whether its optional dependency is loaded.

detect(image, opts)

Detects regions in image for the classes in opts[:classes].

identity(opts)

Returns the detector's cache identity. See "Identity" above.

ready?(opts)

Returns whether detect/2 can run without first downloading model files.

supported_classes(opts)

Returns every class name this detector can produce, in the URL spelling.

warmup(opts)

Loads or downloads models ahead of the first request.

Functions

ready?(module, opts)

Invoke the optional ready?/1 callback if the detector implements it, else available?/1.

warmup(module, opts)

Invoke the optional warmup/1 callback if the detector implements it, else :ok.

Types

region()

@type region() :: %{
  label: String.t(),
  score: float(),
  box: {number(), number(), number(), number()}
}

A detected region of interest.

box is {x, y, width, height} in pixels of the image passed to detect/2, measured from its top-left corner (x grows right, y grows down).

Callbacks

available?(opts)

@callback available?(opts :: keyword()) :: boolean()

Returns whether the detector can run in this build, for example whether its optional dependency is loaded.

detect(image, opts)

@callback detect(image :: Vix.Vips.Image.t(), opts :: keyword()) ::
  {:ok, [region()]} | {:error, term()}

Detects regions in image for the classes in opts[:classes].

Returns {:ok, regions}, where an empty list means nothing was found, or {:error, reason}. See "Regions" and "Errors" above.

identity(opts)

@callback identity(opts :: keyword()) :: {module(), term()}

Returns the detector's cache identity. See "Identity" above.

ready?(opts)

(optional)

@callback ready?(opts :: keyword()) :: boolean()

Returns whether detect/2 can run without first downloading model files.

Optional. Without it, a detector is ready whenever available?/1 is true. See "Availability and readiness" above.

supported_classes(opts)

@callback supported_classes(opts :: keyword()) :: [String.t()]

Returns every class name this detector can produce, in the URL spelling.

Called with [] to route classes and to reject unknown classes with 400. It must not load a model, and must return the full list even when available?/1 is false.

warmup(opts)

(optional)

@callback warmup(opts :: keyword()) :: :ok | {:error, term()}

Loads or downloads models ahead of the first request.

Optional. ImagePipe.Transform.Detector.Warmup calls it with its :opts plus :classes. It returns :ok or {:error, reason}. The worker logs and retries errors.

Functions

ready?(module, opts)

@spec ready?(module(), keyword()) :: boolean()

Invoke the optional ready?/1 callback if the detector implements it, else available?/1.

warmup(module, opts)

@spec warmup(module(), keyword()) :: :ok | {:error, term()}

Invoke the optional warmup/1 callback if the detector implements it, else :ok.

The callback is optional for host-provided detectors, so its presence is checked at this boundary.