Iconvex.Codec behaviour (iconvex v0.1.1)

Copy Markdown View Source

Behaviour implemented by codecs supplied by external libraries.

A codec converts between its byte representation and a list of Unicode code points. decode_discard/1, encode_discard/1, and encode_substitute/2 are separate required callbacks so codecs with shift state or multi-code-point mappings can provide correct linear implementations of conversion policies.

encode_substitute/2 must retain state and longest-match semantics while applying replacements in one pass. decode_error_recovery/0 lets a whole-string transform declare that its byte stream cannot be restarted after a malformed unit. decode_to_utf8/1 and encode_from_utf8/1 are optional strict fast paths. decode_to_ucs4_discard/2 and encode_from_ucs4_discard/2 are optional policy-preserving fast paths for explicit UCS-4 transports. An external library can implement them directly or pass :direct_adapter to use Iconvex.Codec when a shared engine exposes the same callbacks.

One-shot conversions with on_invalid_byte arbitrate each valid source prefix against the target before invoking the next invalid-byte handler. Iconvex uses incremental encoder callbacks when available. Without them it may invoke the required encoder on cumulative prefixes and again for the final result, so external encoder callbacks must be deterministic and side-effect-safe. Implementing encode_chunk/3, or stream_encoder_init/0 plus encode_chunk/4 for a stateful codec, keeps this arbitration linear.

Summary

Callbacks

Additional case-insensitive names. Defaults to an empty list.

Canonical, case-insensitive encoding name.

Stable codec identifier used by shared codec engines. Defaults to the module.

Strictly decodes a complete encoded binary into Unicode code points.

Incrementally decodes a stateless codec.

Incrementally decodes a stateful external codec. Error offsets are relative to input.

Decodes while discarding malformed input using codec-native state handling.

Number of source bytes consumed when one-shot or streaming policy recovers from one decode error. The default is one byte.

Declares whether decode-policy recovery may restart after an invalid unit.

Advances codec-owned decoder state after policy recovery consumes an invalid source unit.

Optional direct discard decoder to explicit UCS-4 byte order.

Optional direct strict decoder to UTF-8.

Strictly encodes Unicode code points.

Incrementally encodes a stateless codec.

Incrementally encodes a stateful external codec without finalizing before final?.

Encodes while discarding unrepresentable code points in one linear pass.

Optional direct discard encoder from aligned explicit UCS-4 byte order.

Optional direct strict encoder from UTF-8.

Encodes in one pass, replacing each unrepresentable code point with replacer.(codepoint).

Declares that the codec carries designation or shift state.

Initial decoder state for a stateful external codec's Stream implementation.

Initial encoder state for a stateful external codec's Stream implementation.

Types

decode_error()

@type decode_error() ::
  {:error, :invalid_sequence | :incomplete_sequence, non_neg_integer(),
   binary()}

decode_result()

@type decode_result() :: {:ok, [non_neg_integer()]} | decode_error()

encode_error()

@type encode_error() :: {:error, :unrepresentable_character, non_neg_integer()}

encode_result()

@type encode_result() :: {:ok, binary()} | encode_error()

legacy_direct_encode_error()

@type legacy_direct_encode_error() ::
  {:encode_error, :unrepresentable_character, non_neg_integer()}

stream_encode_policy()

@type stream_encode_policy() ::
  :error | :discard | {:replace, (non_neg_integer() -> [non_neg_integer()])}

Callbacks

aliases()

(optional)
@callback aliases() :: [String.t()]

Additional case-insensitive names. Defaults to an empty list.

canonical_name()

@callback canonical_name() :: String.t()

Canonical, case-insensitive encoding name.

codec_id()

(optional)
@callback codec_id() :: atom()

Stable codec identifier used by shared codec engines. Defaults to the module.

decode(binary)

@callback decode(binary()) :: decode_result()

Strictly decodes a complete encoded binary into Unicode code points.

decode_chunk(binary, final?)

(optional)
@callback decode_chunk(binary(), final? :: boolean()) ::
  {:ok, [non_neg_integer()], pending :: binary()} | decode_error()

Incrementally decodes a stateless codec.

When final? is false, a valid source prefix that may be extended by a later chunk must be returned as pending instead of decoded prematurely.

decode_chunk(binary, state, final?)

(optional)
@callback decode_chunk(binary(), state :: term(), final? :: boolean()) ::
  {:ok, [non_neg_integer()], next_state :: term(), pending :: binary()}
  | decode_error()

Incrementally decodes a stateful external codec. Error offsets are relative to input.

decode_discard(binary)

@callback decode_discard(binary()) :: {:ok, [non_neg_integer()]} | decode_error()

Decodes while discarding malformed input using codec-native state handling.

decode_error_consumption(arg1, sequence)

(optional)
@callback decode_error_consumption(
  :invalid_sequence | :incomplete_sequence,
  sequence :: binary()
) :: pos_integer()

Number of source bytes consumed when one-shot or streaming policy recovers from one decode error. The default is one byte.

Fixed-width external codecs should return their complete invalid unit width so discard/replacement recovery cannot lose framing. sequence is the exact sequence reported by the codec at the failing offset.

decode_error_recovery()

(optional)
@callback decode_error_recovery() :: :resynchronize | :stop

Declares whether decode-policy recovery may restart after an invalid unit.

The default, :resynchronize, decodes the valid prefix, consumes the codec-declared invalid unit, and continues. Whole-string transforms whose suffix loses meaning after an error return :stop. For :stop, Iconvex obtains the retained output from decode_discard/1, invokes one invalid-byte event, appends any requested replacement, and does not reinterpret the tail.

decode_recovery_state(state, arg2, sequence, consumed)

(optional)
@callback decode_recovery_state(
  state :: term(),
  :invalid_sequence | :incomplete_sequence,
  sequence :: binary(),
  consumed :: binary()
) :: term()

Advances codec-owned decoder state after policy recovery consumes an invalid source unit.

The default is to retain the state returned by decoding the valid prefix. Stateful codecs whose framing includes counters or bounded payloads should update those values for consumed without decoding it as ordinary input.

decode_to_ucs4_discard(binary, arg2)

(optional)
@callback decode_to_ucs4_discard(binary(), :big | :little) :: {:ok, binary()} | :miss

Optional direct discard decoder to explicit UCS-4 byte order.

decode_to_utf8(binary)

(optional)
@callback decode_to_utf8(binary()) :: {:ok, binary()} | decode_error()

Optional direct strict decoder to UTF-8.

encode(list)

@callback encode([non_neg_integer()]) :: encode_result()

Strictly encodes Unicode code points.

encode_chunk(list, final?, stream_encode_policy)

(optional)
@callback encode_chunk(
  [non_neg_integer()],
  final? :: boolean(),
  stream_encode_policy()
) :: {:ok, binary(), pending :: [non_neg_integer()]} | encode_error()

Incrementally encodes a stateless codec.

When final? is false, a Unicode suffix that may participate in a longer destination mapping must be returned as pending.

encode_chunk(list, state, final?, stream_encode_policy)

(optional)
@callback encode_chunk(
  [non_neg_integer()],
  state :: term(),
  final? :: boolean(),
  stream_encode_policy()
) ::
  {:ok, binary(), next_state :: term(), pending :: [non_neg_integer()]}
  | encode_error()

Incrementally encodes a stateful external codec without finalizing before final?.

encode_discard(list)

@callback encode_discard([non_neg_integer()]) :: encode_result()

Encodes while discarding unrepresentable code points in one linear pass.

encode_from_ucs4_discard(binary, arg2)

(optional)
@callback encode_from_ucs4_discard(binary(), :big | :little) :: {:ok, binary()} | :miss

Optional direct discard encoder from aligned explicit UCS-4 byte order.

encode_from_utf8(binary)

(optional)
@callback encode_from_utf8(binary()) ::
  encode_result()
  | legacy_direct_encode_error()
  | {:decode_error, :invalid_sequence | :incomplete_sequence, non_neg_integer(),
     binary()}

Optional direct strict encoder from UTF-8.

New codecs should return the ordinary encode_error() form for an unrepresentable character. The destination-tagged {:encode_error, :unrepresentable_character, codepoint} form remains accepted for backward compatibility with existing direct fast paths; Iconvex normalizes it without invoking encode/1 as a fallback.

encode_substitute(list, function)

@callback encode_substitute([non_neg_integer()], (non_neg_integer() ->
                                          [non_neg_integer()])) ::
  encode_result()

Encodes in one pass, replacing each unrepresentable code point with replacer.(codepoint).

stateful?()

(optional)
@callback stateful?() :: boolean()

Declares that the codec carries designation or shift state.

stream_decoder_init()

(optional)
@callback stream_decoder_init() :: term()

Initial decoder state for a stateful external codec's Stream implementation.

stream_encoder_init()

(optional)
@callback stream_encoder_init() :: term()

Initial encoder state for a stateful external codec's Stream implementation.