Emily.MixedPrecision.LossScaler (emily v0.3.2)

Copy Markdown View Source

Dynamic loss-scaler state for mixed-precision training.

Tracks the current scale factor and the number of consecutive successful (non-overflow) steps. On overflow the scale is halved; after growth_interval successful steps it doubles.

Summary

Functions

new(opts \\ [])

Create a new loss scaler.

Functions

new(opts \\ [])

Create a new loss scaler.

Examples

iex> scaler = Emily.MixedPrecision.LossScaler.new()
iex> scaler.scale
65536.0

iex> scaler = Emily.MixedPrecision.LossScaler.new(scale: 32.0, min_scale: 1.0)
iex> {scaler.scale, scaler.min_scale}
{32.0, 1.0}