Ftfy.Badness (ftfy v0.1.0)

Copy Markdown View Source

A heuristic that detects likely mojibake. It signals which segments of text need to be fixed, and when fixing can stop.

Port of ftfy.badness. The big BADNESS_RE pattern itself is generated from the reference implementation (see the badness_pattern/0 function in the generated Ftfy.Data module).

Summary

Functions

badness(text)

Count the unlikely character sequences in text. A value greater than 0 indicates some of it seems to be mojibake.

badness_re()
is_bad(text)

Whether text looks like it contains mojibake. Faster than badness/1 because it stops at the first match.

Functions

badness(text)

@spec badness(binary()) :: non_neg_integer()

Count the unlikely character sequences in text. A value greater than 0 indicates some of it seems to be mojibake.

badness_re()

is_bad(text)

@spec is_bad(binary()) :: boolean()

Whether text looks like it contains mojibake. Faster than badness/1 because it stops at the first match.