Behaviour implemented by codecs supplied by external libraries.
A codec converts between its byte representation and a list of Unicode code
points. decode_discard/1, encode_discard/1, and encode_substitute/2 are
separate required callbacks so codecs with shift state or multi-code-point
mappings can provide correct linear implementations of conversion policies.
encode_substitute/2 must retain state and longest-match semantics while
applying replacements in one pass.
decode_error_recovery/0 lets a whole-string transform declare that its
byte stream cannot be restarted after a malformed unit.
decode_to_utf8/1 and encode_from_utf8/1 are optional strict fast paths.
decode_to_ucs4_discard/2 and encode_from_ucs4_discard/2 are optional
policy-preserving fast paths for explicit UCS-4 transports. An external
library can implement them directly or pass :direct_adapter to
use Iconvex.Codec when a shared engine exposes the same callbacks.
One-shot conversions with on_invalid_byte arbitrate each valid source
prefix against the target before invoking the next invalid-byte handler.
Iconvex uses incremental encoder callbacks when available. Without them it
may invoke the required encoder on cumulative prefixes and again for the
final result, so external encoder callbacks must be deterministic and
side-effect-safe. Implementing encode_chunk/3, or
stream_encoder_init/0 plus encode_chunk/4 for a stateful codec, keeps
this arbitration linear.
Summary
Callbacks
Additional case-insensitive names. Defaults to an empty list.
Canonical, case-insensitive encoding name.
Stable codec identifier used by shared codec engines. Defaults to the module.
Strictly decodes a complete encoded binary into Unicode code points.
Incrementally decodes a stateless codec.
Incrementally decodes a stateful external codec. Error offsets are relative to input.
Decodes while discarding malformed input using codec-native state handling.
Number of source bytes consumed when one-shot or streaming policy recovers from one decode error. The default is one byte.
Declares whether decode-policy recovery may restart after an invalid unit.
Advances codec-owned decoder state after policy recovery consumes an invalid source unit.
Optional direct discard decoder to explicit UCS-4 byte order.
Optional direct strict decoder to UTF-8.
Strictly encodes Unicode code points.
Incrementally encodes a stateless codec.
Incrementally encodes a stateful external codec without finalizing before final?.
Encodes while discarding unrepresentable code points in one linear pass.
Optional direct discard encoder from aligned explicit UCS-4 byte order.
Optional direct strict encoder from UTF-8.
Encodes in one pass, replacing each unrepresentable code point with replacer.(codepoint).
Declares that the codec carries designation or shift state.
Initial decoder state for a stateful external codec's Stream implementation.
Initial encoder state for a stateful external codec's Stream implementation.
Types
@type decode_error() :: {:error, :invalid_sequence | :incomplete_sequence, non_neg_integer(), binary()}
@type decode_result() :: {:ok, [non_neg_integer()]} | decode_error()
@type encode_error() :: {:error, :unrepresentable_character, non_neg_integer()}
@type encode_result() :: {:ok, binary()} | encode_error()
@type legacy_direct_encode_error() :: {:encode_error, :unrepresentable_character, non_neg_integer()}
@type stream_encode_policy() :: :error | :discard | {:replace, (non_neg_integer() -> [non_neg_integer()])}
Callbacks
@callback aliases() :: [String.t()]
Additional case-insensitive names. Defaults to an empty list.
@callback canonical_name() :: String.t()
Canonical, case-insensitive encoding name.
@callback codec_id() :: atom()
Stable codec identifier used by shared codec engines. Defaults to the module.
@callback decode(binary()) :: decode_result()
Strictly decodes a complete encoded binary into Unicode code points.
@callback decode_chunk(binary(), final? :: boolean()) :: {:ok, [non_neg_integer()], pending :: binary()} | decode_error()
Incrementally decodes a stateless codec.
When final? is false, a valid source prefix that may be extended by a
later chunk must be returned as pending instead of decoded prematurely.
@callback decode_chunk(binary(), state :: term(), final? :: boolean()) :: {:ok, [non_neg_integer()], next_state :: term(), pending :: binary()} | decode_error()
Incrementally decodes a stateful external codec. Error offsets are relative to input.
@callback decode_discard(binary()) :: {:ok, [non_neg_integer()]} | decode_error()
Decodes while discarding malformed input using codec-native state handling.
@callback decode_error_consumption( :invalid_sequence | :incomplete_sequence, sequence :: binary() ) :: pos_integer()
Number of source bytes consumed when one-shot or streaming policy recovers from one decode error. The default is one byte.
Fixed-width external codecs should return their complete invalid unit width
so discard/replacement recovery cannot lose framing. sequence is the exact
sequence reported by the codec at the failing offset.
@callback decode_error_recovery() :: :resynchronize | :stop
Declares whether decode-policy recovery may restart after an invalid unit.
The default, :resynchronize, decodes the valid prefix, consumes the
codec-declared invalid unit, and continues. Whole-string transforms whose
suffix loses meaning after an error return :stop. For :stop, Iconvex
obtains the retained output from decode_discard/1, invokes one invalid-byte
event, appends any requested replacement, and does not reinterpret the tail.
@callback decode_recovery_state( state :: term(), :invalid_sequence | :incomplete_sequence, sequence :: binary(), consumed :: binary() ) :: term()
Advances codec-owned decoder state after policy recovery consumes an invalid source unit.
The default is to retain the state returned by decoding the valid prefix.
Stateful codecs whose framing includes counters or bounded payloads should
update those values for consumed without decoding it as ordinary input.
Optional direct discard decoder to explicit UCS-4 byte order.
@callback decode_to_utf8(binary()) :: {:ok, binary()} | decode_error()
Optional direct strict decoder to UTF-8.
@callback encode([non_neg_integer()]) :: encode_result()
Strictly encodes Unicode code points.
@callback encode_chunk( [non_neg_integer()], final? :: boolean(), stream_encode_policy() ) :: {:ok, binary(), pending :: [non_neg_integer()]} | encode_error()
Incrementally encodes a stateless codec.
When final? is false, a Unicode suffix that may participate in a longer
destination mapping must be returned as pending.
@callback encode_chunk( [non_neg_integer()], state :: term(), final? :: boolean(), stream_encode_policy() ) :: {:ok, binary(), next_state :: term(), pending :: [non_neg_integer()]} | encode_error()
Incrementally encodes a stateful external codec without finalizing before final?.
@callback encode_discard([non_neg_integer()]) :: encode_result()
Encodes while discarding unrepresentable code points in one linear pass.
Optional direct discard encoder from aligned explicit UCS-4 byte order.
@callback encode_from_utf8(binary()) :: encode_result() | legacy_direct_encode_error() | {:decode_error, :invalid_sequence | :incomplete_sequence, non_neg_integer(), binary()}
Optional direct strict encoder from UTF-8.
New codecs should return the ordinary encode_error() form for an
unrepresentable character. The destination-tagged
{:encode_error, :unrepresentable_character, codepoint} form remains
accepted for backward compatibility with existing direct fast paths; Iconvex
normalizes it without invoking encode/1 as a fallback.
@callback encode_substitute([non_neg_integer()], (non_neg_integer() -> [non_neg_integer()])) :: encode_result()
Encodes in one pass, replacing each unrepresentable code point with replacer.(codepoint).
@callback stateful?() :: boolean()
Declares that the codec carries designation or shift state.
@callback stream_decoder_init() :: term()
Initial decoder state for a stateful external codec's Stream implementation.
@callback stream_encoder_init() :: term()
Initial encoder state for a stateful external codec's Stream implementation.