segmenter

Locale-aware text segmentation via the JavaScript Intl.Segmenter. Reusing a built Segmenter across many calls is significantly faster than building one per call.

Types

Builder

opaque </>

The configuration for a Segmenter.

pub opaque type Builder

Granularity

</>

The unit of segmentation. Maps the JavaScript granularity option.

pub type Granularity {
  Grapheme
  Word
  Sentence
}

Constructors

```
Grapheme
```
Split into user-perceived characters — accounts for combining marks, emoji sequences, and the like (the default).
```
Word
```
Split into words.
```
Sentence
```
Split into sentences.

ResolvedOptions

</>

The options the runtime resolved for a Segmenter, including the defaults it filled in. locale is the BCP 47 tag chosen from the requested priority list (e.g., "en-US").

pub type ResolvedOptions {
  ResolvedOptions(locale: String, granularity: Granularity)
}

Constructors

ResolvedOptions(locale: String, granularity: Granularity)

A single segment from a segmented string. value is the substring, index is the UTF-16 code-unit position within the input, and word_like is Some only when the segmenter was configured with Word granularity.

pub type Segment {
  Segment(
    value: String,
    index: Int,
    word_like: option.Option(Bool),
  )
}

Constructors

Segment(
  value: String,
  index: Int,
  word_like: option.Option(Bool),
)

Segmenter

</>

A configured segmenter that splits a string into graphemes, words, or sentences using locale-specific rules.

See Intl.Segmenter on MDN.

pub type Segmenter

Values

build

</>

pub fn build(builder: Builder) -> Result(Segmenter, Nil)

Constructs a Segmenter from the configured builder. Returns Error(Nil) if any locale tag is structurally invalid.

containing

</>

pub fn containing(
  segmenter: Segmenter,
  in input: String,
  at_index index: Int,
) -> Result(Segment, Nil)

Returns the Segment containing the UTF-16 code-unit at index, or Error(Nil) if index falls outside input.

new

</>

pub fn new(locales: List(String)) -> Builder

Creates a Builder for the given locale priority list. The runtime picks the first locale it supports; pass an empty list to use the runtime’s default locale.

resolved_options

</>

pub fn resolved_options(segmenter: Segmenter) -> ResolvedOptions

The locale and granularity the runtime resolved from the builder’s configuration.

segment

</>

pub fn segment(
  segmenter: Segmenter,
  input: String,
) -> yielder.Yielder(Segment)

Splits input into segments according to the segmenter’s configured Granularity. The returned yielder produces each Segment lazily, in order.

supported_locales_of

</>

pub fn supported_locales_of(
  locales: List(String),
) -> Result(List(String), Nil)

Filters locales to those the runtime supports for segmentation, preserving the input order. Returns Error(Nil) if any locale tag is structurally malformed.

with_granularity

</>

pub fn with_granularity(
  builder: Builder,
  granularity: Granularity,
) -> Builder

Sets the unit of segmentation.

with_locale_matcher

</>

pub fn with_locale_matcher(
  builder: Builder,
  locale_matcher: intl.LocaleMatcher,
) -> Builder

Sets the locale-matching algorithm used to pick a locale from the priority list.