LemonAi.Text (lemon_ai v0.1.0)

View Source

Generic text truncation primitives.

The helpers here preserve UTF-8 validity for byte-based head and middle truncation. Callers still own domain-specific markers, notices, metadata, and thresholds.

Summary

Functions

Truncates text to a UTF-8-safe byte prefix and appends a marker.

Truncates text by Unicode characters and appends a marker.

Keeps a UTF-8-safe head and tail within max_bytes, inserting a marker.

Truncates content to the last lines and bytes, returning metadata.

Functions

truncate_bytes_utf8(text, max_bytes, opts \\ [])

@spec truncate_bytes_utf8(String.t(), non_neg_integer(), keyword()) :: String.t()

Truncates text to a UTF-8-safe byte prefix and appends a marker.

The marker may be a binary or a function receiving the removed byte count. Pass marker: "" for markerless truncation. This keeps the byte-size check in bytes, unlike truncate_chars/3.

truncate_chars(text, max, opts \\ [])

@spec truncate_chars(String.t(), non_neg_integer(), keyword()) :: String.t()

Truncates text by Unicode characters and appends a marker.

Options:

  • :marker - marker appended after the kept prefix, default "...".
  • :force - append the marker even when the text is not longer than max.

truncate_middle_utf8(text, max_bytes, opts \\ [])

@spec truncate_middle_utf8(String.t(), integer(), keyword()) :: String.t()

Keeps a UTF-8-safe head and tail within max_bytes, inserting a marker.

Options:

  • :marker - marker binary or function receiving removed byte count.
  • :marker_reserve - byte budget reserved before splitting head/tail.
  • :head_percent - percentage of the remaining budget used for the head.

truncate_tail(content, opts \\ [])

@spec truncate_tail(
  String.t(),
  keyword()
) :: {String.t(), boolean(), map()}

Truncates content to the last lines and bytes, returning metadata.

Options:

  • :max_bytes - maximum bytes to keep before adding the notice.
  • :max_lines - maximum lines to keep.
  • :notice - function receiving {total_lines, total_bytes}.