defmodule Difflib.SequenceMatcher do import While import Counter counter([:besti, :bestj, :bestsize]) @moduledoc """ SequenceMatcher is a flexible module for comparing pairs of sequences of any type, so long as the sequence elements are hashable. ## The algorithm The basic algorithm predates, and is a little fancier than, an algorithm published in the late 1980's by Ratcliff and Obershelp under the hyperbolic name "gestalt pattern matching". The basic idea is to find the longest contiguous matching subsequence that contains no "junk" elements (R-O doesn't address junk). The same idea is then applied recursively to the pieces of the sequences to the left and to the right of the matching subsequence. This does not yield minimal edit sequences, but does tend to yield matches that "look right" to people. SequenceMatcher tries to compute a "human-friendly diff" between two sequences. Unlike e.g. UNIX(tm) diff, the fundamental notion is the longest *contiguous* & junk-free matching subsequence. That's what catches peoples' eyes. The Windows(tm) windiff has another interesting notion, pairing up elements that appear uniquely in each sequence. That, and the method here, appear to yield more intuitive difference reports than does diff. This method appears to be the least vulnerable to synching up on blocks of "junk lines", though (like blank lines in ordinary text files, or maybe "

" lines in HTML files). That may be because this is the only method of the 3 that has a *concept* of "junk" . ## Examples Example, comparing two strings, and considering blanks to be "junk" iex> is_junk = fn c -> c == " " end iex> a = "private Thread currentThread;" iex> b = "private volatile Thread currentThread;" iex> SequenceMatcher.ratio(a, b, is_junk: is_junk) 0.8656716417910447 `ratio/3` returns a float between 0 and 1, measuring the "similarity" of the sequences. As a rule of thumb, a `ratio/3` value over 0.6 means the sequences are close matches. If you're only interested in where the sequences match, `get_matching_blocks/3` is handy: iex> for {a, b, size} <- SequenceMatcher.get_matching_blocks(a, b, is_junk: is_junk) do iex> IO.puts("a[\#{a}] and b[\#{b}] match for \#{size} elements") iex> end a[0] and b[0] match for 8 elements a[8] and b[17] match for 21 elements a[29] and b[38] match for 0 elements Note that the last tuple returned by `get_matching_blocks/3` is always a dummy, `{length(a), length(b), 0}`, and this is the only case in which the last tuple element (number of elements matched) is 0. If you want to know how to change the first sequence into the second, use `get_opcodes/3` iex> for {op, a1, a2, b1, b2} <- SequenceMatcher.get_opcodes(a, b, is_junk: is_junk) do iex> IO.puts("\#{op} a[\#{a1}:\#{a2}] b[\#{b1}:\#{b2}]") iex> end equal a[0:8] b[0:8] insert a[8:8] b[8:17] equal a[8:29] b[17:38] See also function `get_close_matches/3` in this module, which shows how simple code building on SequenceMatcher can be used to do useful work. Timing: Basic R-O is cubic time worst case and quadratic time expected case. SequenceMatcher is quadratic time for the worst case and has expected-case behavior dependent in a complicated way on how many elements the sequences have in common; best case time is linear. """ @doc """ Analyzes an input for junk elements. ## Background Because `is_junk` is a user-defined function, and we test for junk a LOT, it's important to minimize the number of calls. Before the tricks described here, `chain_b/2` was by far the most time-consuming routine in the whole module! If anyone sees Jim Roskind, thank him again for [profile.py](https://github.com/python/cpython/blob/main/Lib/profile.py) -- I never would have guessed that. The first trick is to build `b2j` ignoring the possibility of junk. I.e., we don't call is_junk at all yet. Throwing out the junk later is much cheaper than building `b2j` "right" from the start. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Example iex> is_junk = fn x -> x == " " end iex> b = "abcd abcd" iex> SequenceMatcher.chain_b(b, is_junk: is_junk) %{ b2j: %{ "a" => [0, 5], "b" => [1, 6], "c" => [2, 7], "d" => [3, 8] }, isbjunk: #Function<1.118419402/1>, isbpopular: #Function<1.118419402/1>, bjunk: %{" " => true}, bpopular: %{} } """ def chain_b(b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:is_junk, auto_junk: true]) is_junk = Keyword.get(validated_opts, :is_junk) auto_junk = Keyword.get(validated_opts, :auto_junk) vals = get_vals(b) b2j = Enum.with_index(vals) |> Enum.reduce(%{}, fn {elt, i}, acc -> indices = Map.get(acc, elt, []) Map.put(acc, elt, indices ++ [i]) end) # Purge junk elements result = if is_nil(is_junk) do %{ junk: %{}, b2j: b2j } else Map.keys(b2j) |> Enum.reduce(%{junk: %{}, b2j: b2j}, fn elt, acc -> if is_junk.(elt) do %{ junk: Map.put(acc.junk, elt, true), b2j: Map.delete(acc.b2j, elt) } else acc end end) end # Purge popular elements that are not junk popular = %{} n = length(vals) next_result = if auto_junk and n >= 200 do ntest = Float.floor(n / 100) + 1 Map.to_list(result.b2j) |> Enum.reduce(%{popular: popular, b2j: result.b2j}, fn {elt, idxs}, acc -> if length(idxs) > ntest do %{ popular: Map.put(acc.popular, elt, true), b2j: Map.delete(acc.b2j, elt) } else acc end end) else %{ popular: popular, b2j: result.b2j } end # Now for x in b, isjunk.(x) == x in junk, but the latter is much faster. # Since the number of *unique* junk elements is probably small, the # memory burden of keeping this set alive is likely trivial compared to # the size of b2j. isbjunk = fn b -> Map.has_key?(result.junk, b) end isbpopular = fn b -> Map.has_key?(next_result.popular, b) end %{ b2j: next_result.b2j, isbjunk: isbjunk, isbpopular: isbpopular, bjunk: result.junk, bpopular: next_result.popular } end @doc """ Find longest matching block in a[alo...ahi] and b[blo...bhi]. ## Description If `is_junk` is not defined: Return `{i,j,k}` such that `a[i...i+k]` is equal to `b[j...j+k]`, where alo <= i <= i+k <= ahi blo <= j <= j+k <= bhi and for all `{i',j',k'}` meeting those conditions, k >= k' i <= i' and if i == i', j <= j' In other words, of all maximal matching blocks, return one that starts earliest in a, and of all those maximal matching blocks that start earliest in a, return the one that starts earliest in b. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `alo` - Optional parameter alo is the lower bound of the range in a to consider. The default is 0. - `ahi` - Optional parameter ahi is the upper bound of the range in a to consider. The default is length of a. - `blo` - Optional parameter blo is the lower bound of the range in b to consider. The default is 0. - `bhi` - Optional parameter bhi is the upper bound of the range in b to consider. The default is length of b. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Examples iex> is_junk = fn x -> x == " " end iex> a = " abcd" iex> b = "abcd abcd" iex> SequenceMatcher.find_longest_match(a, b, alo: 0, ahi: 5, blo: 0, bhi: 9, is_junk: is_junk) {1, 0, 4} iex> a = "ab" iex> b = "c" iex> SequenceMatcher.find_longest_match(a, b, alo: 0, ahi: 2, blo: 0, bhi: 1) {0, 0, 0} ## CAUTION CAUTION: stripping common prefix or suffix would be incorrect. E.g., ab acab Longest matching block is "ab", but if common prefix is stripped, it's "a" (tied with "b"). UNIX(tm) diff does so strip, so ends up claiming that ab is changed to acab by inserting "ca" in the middle. That's minimal but unintuitive: "it's obvious" that someone inserted "ac" at the front. Windiff ends up at the same place as diff, but by pairing up the unique 'b's and then matching the first two 'a's. """ def find_longest_match(a, b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:ahi, :bhi, :is_junk, alo: 0, blo: 0, auto_junk: true]) alo = Keyword.get(validated_opts, :alo) blo = Keyword.get(validated_opts, :blo) ahi = Keyword.get(validated_opts, :ahi) bhi = Keyword.get(validated_opts, :bhi) %{b2j: b2j, isbjunk: isbjunk} = chain_b(b, Keyword.drop(validated_opts, [:alo, :ahi, :blo, :bhi])) ref = :counters.new(3, [:atomics]) set_besti(ref, alo) set_bestj(ref, blo) set_bestsize(ref, 0) ahi = if is_nil(ahi), do: String.length(a), else: ahi bhi = if is_nil(bhi), do: String.length(b), else: bhi a_at = if is_binary(a), do: &String.at(a, &1), else: &Enum.at(a, &1) b_at = if is_binary(b), do: &String.at(b, &1), else: &Enum.at(b, &1) # find longest junk-free match # during an iteration of the loop, j2len[j] = length of longest # junk-free match ending with a[i-1] and b[j] Enum.reduce(alo..(ahi - 1), %{}, fn i, j2len -> # look at all instances of a[i] in b; note that because # b2j has no junk keys, the loop is skipped if a[i] is junk Map.get(b2j, a_at.(i), []) |> Enum.reduce_while(%{}, fn j, newj2len -> cond do j < blo -> {:cont, newj2len} j >= bhi -> {:halt, newj2len} true -> k = Map.get(j2len, j - 1, 0) + 1 if k > bestsize(ref) do set_besti(ref, i - k + 1) set_bestj(ref, j - k + 1) set_bestsize(ref, k) end {:cont, Map.put(newj2len, j, k)} end end) end) # Extend the best by non-junk elements on each end. In particular, # "popular" non-junk elements aren't in b2j, which greatly speeds # the inner loop above, but also means "the best" match so far # doesn't contain any junk *or* popular non-junk elements. # max_iterations = min(results.besti - alo, results.bestj - blo) while besti(ref) > alo and bestj(ref) > blo and not isbjunk.(b_at.(bestj(ref) - 1)) and a_at.(besti(ref) - 1) == b_at.(bestj(ref) - 1) do dec_besti(ref) dec_bestj(ref) inc_bestsize(ref) end while besti(ref) + bestsize(ref) < ahi and bestj(ref) + bestsize(ref) < bhi and not isbjunk.(b_at.(bestj(ref) + bestsize(ref))) and a_at.(besti(ref) + bestsize(ref)) == b_at.(bestj(ref) + bestsize(ref)) do inc_bestsize(ref) end # Now that we have a wholly interesting match (albeit possibly # empty!), we may as well suck up the matching junk on each # side of it too. Can't think of a good reason not to, and it # saves post-processing the (possibly considerable) expense of # figuring out what to do with it. In the case of an empty # interesting match, this is clearly the right thing to do, # because no other kind of match is possible in the regions. while besti(ref) > alo and bestj(ref) > blo and isbjunk.(b_at.(bestj(ref) - 1)) and a_at.(besti(ref) - 1) == b_at.(bestj(ref) - 1) do dec_besti(ref) dec_bestj(ref) inc_bestsize(ref) end while besti(ref) + bestsize(ref) < ahi and bestj(ref) + bestsize(ref) < bhi and isbjunk.(b_at.(bestj(ref) + bestsize(ref))) and a_at.(besti(ref) + bestsize(ref)) == b_at.(bestj(ref) + bestsize(ref)) do inc_bestsize(ref) end {besti(ref), bestj(ref), bestsize(ref)} end @doc """ Return list of triples describing matching subsequences. ## Description Each triple is of the form `{i, j, n}`, and means that `a[i...i+n] == b[j...j+n]`. The triples are monotonically increasing in `i` and in `j`. it's also guaranteed that if `{i, j, n}` and `{i', j', n'}` are adjacent triples in the list, and the second is not the last triple in the list, then `i+n != i'` or `j+n != j'`. IOW, adjacent triples never describe adjacent equal blocks. The last triple is a dummy, `{a.length, b.length, 0}`, and is the only triple with `n==0`. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Example iex> a = "abxcd" iex> b = "abcd" iex> SequenceMatcher.get_matching_blocks(a, b) [{0, 0, 2}, {3, 2, 2}, {5, 4, 0}] """ def get_matching_blocks(a, b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:is_junk, auto_junk: true]) is_junk = Keyword.get(validated_opts, :is_junk) auto_junk = Keyword.get(validated_opts, :auto_junk) la = get_length(a) lb = get_length(b) # This is most naturally expressed as a recursive algorithm, but # at least one user bumped into extreme use cases that exceeded # the recursion limit on their box. So, now we maintain a list # ('queue`) of blocks we still need to look at, and append partial # results to `matching_blocks` in a loop; the matches are sorted # at the end. Loop an aribtrary number of times matching_blocks = reduce_while(%{queue: [{0, la, 0, lb}], mb: []}, fn %{queue: queue, mb: mb} -> if queue == [] do {:halt, mb} else [{alo, ahi, blo, bhi} | next_queue] = queue {i, j, k} = find_longest_match(a, b, alo: alo, ahi: ahi, blo: blo, bhi: bhi, is_junk: is_junk, auto_junk: auto_junk ) x = {i, j, k} if k != 0 do next_queue = if alo < i and blo < j do [{alo, i, blo, j}] ++ next_queue else next_queue end next_queue = if i + k < ahi and j + k < bhi do [{i + k, ahi, j + k, bhi}] ++ next_queue else next_queue end {:cont, %{queue: next_queue, mb: [x] ++ mb}} else {:cont, %{queue: next_queue, mb: mb}} end end end) |> Enum.sort() # It's possible that we have adjacent equal blocks in the # matching_blocks list now. %{non_adjacent: non_adjacent, i1: i1, j1: j1, k1: k1} = matching_blocks |> Enum.reduce(%{non_adjacent: [], i1: 0, j1: 0, k1: 0}, fn {i2, j2, k2}, acc -> # Is this block adjacent to i1, j1, k1? if acc.i1 + acc.k1 == i2 and acc.j1 + acc.k1 == j2 do # Yes, so collapse them -- this just increases the length of # the first block by the length of the second, and the first # block so lengthened remains the block to compare against. Map.put(acc, :k1, acc.k1 + k2) else # Not adjacent. Remember the first block (k1==0 means it's # the dummy we started with), and make the second block the # new block to compare against. next_non_adjacent = if acc.k1 > 0 do acc.non_adjacent ++ [{acc.i1, acc.j1, acc.k1}] else acc.non_adjacent end %{ i1: i2, j1: j2, k1: k2, non_adjacent: next_non_adjacent } end end) final_non_adjacent = non_adjacent ++ if k1 > 0, do: [{i1, j1, k1}], else: [] final_non_adjacent ++ [{la, lb, 0}] end @doc """ Return list of 5-tuples describing how to turn a into b. ## Description Each tuple is of the form `{tag, i1, i2, j1, j2}`. The first tuple has `i1 == j1 == 0`, and remaining tuples have `i1 ==` the `i2` from the tuple preceding it, and likewise for `j1 ==` the previous `j2`. The tags are atoms, with these meanings: - `:replace` - `a[i1...i2]` should be replaced by `b[j1...j2]` - `:delete` - `a[i1...i2]` should be deleted. Note that `j1==j2` in this case. - `:insert` - `b[j1...j2]` should be inserted at `a[i1...i1]`. Note that `i1==i2` in this case. - `:equal` - `a[i1...i2] == b[j1...j2]` ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Example iex> a = "qabxcd" iex> b = "abycdf" iex> SequenceMatcher.get_opcodes(a, b) [ {:delete, 0, 1, 0, 0}, {:equal, 1, 3, 0, 2}, {:replace, 3, 4, 2, 3}, {:equal, 4, 6, 3, 5}, {:insert, 6, 6, 5, 6} ] """ def get_opcodes(a, b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:is_junk, auto_junk: true]) get_matching_blocks(a, b, validated_opts) |> Enum.reduce(%{answer: [], i: 0, j: 0}, fn {ai, bj, size}, acc -> # invariant: we've pumped out correct diffs to change # a[0...i] into b[0...j], and the next matching block is # a[ai...ai+size] == b[bj...bj+size]. So we need to pump # out a diff to change a[i:ai] into b[j...bj], pump out # the matching block, and move [i,j] beyond the match tag = cond do acc.i < ai and acc.j < bj -> :replace acc.i < ai -> :delete acc.j < bj -> :insert true -> nil end next_answer = acc.answer ++ if is_nil(tag), do: [], else: [{tag, acc.i, ai, acc.j, bj}] # the list of matching blocks is terminated by a # sentinel with size 0 final_answer = next_answer ++ if size > 0, do: [{:equal, ai, ai + size, bj, bj + size}], else: [] %{answer: final_answer, i: ai + size, j: bj + size} end) |> Map.get(:answer) end @doc """ Isolate change clusters by eliminating ranges with no changes. ## Description Return a list groups with up to `n` lines of context. Each group is in the same format as returned by `get_opcodes/3`. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `n` - Optional parameter n is the number of lines of context to include in each group. The default is 3. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Example iex> a = Enum.map(1..39, &Integer.to_string/1) iex> b = Enum.slice(a, 0..-1) iex> b = Enum.slice(b, 0..7) ++ ["i"] ++ Enum.slice(b, 8..-1) iex> b = Enum.slice(b, 0..19) ++ ["20x"] ++ Enum.slice(b, 21..-1) iex> b = Enum.slice(b, 0..22) ++ Enum.slice(b, 28..-1) iex> b = Enum.slice(b, 0..29) ++ ["35y"] ++ Enum.slice(b, 31..-1) iex> SequenceMatcher.get_grouped_opcodes(a, b) [ [ {:equal, 5, 8, 5, 8}, {:insert, 8, 8, 8, 9}, {:equal, 8, 11, 9, 12}], [ {:equal, 16, 19, 17, 20}, {:replace, 19, 20, 20, 21}, {:equal, 20, 22, 21, 23}, {:delete, 22, 27, 23, 23}, {:equal, 27, 30, 23, 26} ], [ {:equal, 31, 34, 27, 30}, {:replace, 34, 35, 30, 31}, {:equal, 35, 38, 31, 34} ] ] """ def get_grouped_opcodes(a, b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:is_junk, auto_junk: true, n: 3]) n = Keyword.get(validated_opts, :n) opcodes = get_opcodes(a, b, Keyword.drop(validated_opts, [:n])) codes = if(length(opcodes) > 0, do: opcodes, else: [{:equal, 0, 1, 0, 1}]) # Fixup leading and trailing groups if they show no changes. tag_val = codes |> Enum.at(0) |> elem(0) codes = if tag_val == :equal do {tag, i1, i2, j1, j2} = Enum.at(codes, 0) List.replace_at(codes, 0, {tag, max(i1, i2 - n), i2, max(j1, j2 - n), j2}) else codes end tag_val = codes |> Enum.at(-1) |> elem(0) codes = if tag_val == :equal do {tag, i1, i2, j1, j2} = Enum.at(codes, -1) List.replace_at(codes, -1, {tag, i1, min(i2, i1 + n), j1, min(j2, j1 + n)}) else codes end nn = n + n %{groups: final_groups, group: final_group} = Enum.reduce(codes, %{groups: [], group: []}, fn {tag, i1, i2, j1, j2}, %{groups: groups, group: group} -> # End the current group and start a new one whenever # there is a large range with no changes. {next_acc, nextij} = if tag == :equal and i2 - i1 > nn do group = group ++ [{tag, i1, min(i2, i1 + n), j1, min(j2, j1 + n)}] groups = groups ++ [group] {%{groups: groups, group: []}, %{i1: max(i1, i2 - n), j1: max(j1, j2 - n)}} else {%{groups: groups, group: group}, %{i1: i1, j1: j1}} end updated_group = next_acc.group ++ [{tag, nextij.i1, i2, nextij.j1, j2}] %{ groups: next_acc.groups, group: updated_group } end) if length(final_group) > 0 and not (length(final_group) == 1 and final_group |> Enum.at(0) |> elem(0) == :equal) do final_groups ++ [final_group] else final_groups end end @doc """ Return a measure of the sequences' similarity (float between 0 and 1). ## Description Where `T` is the total number of elements in both sequences, and `M` is the number of matches, this is `2.0*M / T`. Note that this is 1 if the sequences are identical, and 0 if they have nothing in common. `ratio/3` is expensive to compute if you haven't already computed `get_matching_blocks/3` or `get_opcodes/3`, in which case you may want to try `quick_ratio/3` or `real_quick_ratio/3` first to get an upper bound. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Example iex> a = "abcd" iex> b = "bcde" iex> SequenceMatcher.ratio(a, b) 0.75 """ def ratio(a, b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:is_junk, auto_junk: true]) matching_blocks = get_matching_blocks(a, b, validated_opts) matches = Enum.reduce(matching_blocks, 0, fn {_, _, size}, acc -> acc + size end) la = get_length(a) lb = get_length(b) calculate_ratio(matches, la + lb) end @doc """ Return an upper bound on `ratio/3` relatively quickly. ## Description This isn't defined beyond that it is an upper bound on `ratio/3`, and is faster to compute. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. - `opts` - Keyword list of options. - `fullbcount` - Optional parameter fullbcount is a map of the counts of each element in b. It will be constructed if it does not exist. The default is `nil`. ## Example iex> a = "abcd" iex> b = "bcde" iex> SequenceMatcher.quick_ratio(a, b) 0.75 """ def quick_ratio(a, b, opts \\ []) do validated_opts = Keyword.validate!(opts, [:fullbcount]) fullbcount = Keyword.get(validated_opts, :fullbcount) # viewing a and b as multisets, set matches to the cardinality # of their intersection; this counts the number of matches # without regard to order, so is clearly an upper bound a_vals = get_vals(a) b_vals = get_vals(b) fullbcount = if is_nil(fullbcount) do Enum.reduce(b_vals, %{}, fn elt, acc -> Map.put(acc, elt, Map.get(acc, elt, 0) + 1) end) else fullbcount end # avail[x] is the number of times x appears in 'b' less the # number of times we've seen it in 'a' so far ... kinda %{matches: matches} = Enum.reduce(a_vals, %{matches: 0, avail: %{}}, fn elt, acc -> numb = if Map.has_key?(acc.avail, elt) do Map.get(acc.avail, elt) else Map.get(fullbcount, elt, 0) end next_avail = Map.put(acc.avail, elt, numb - 1) next_matches = if numb > 0, do: acc.matches + 1, else: acc.matches %{ avail: next_avail, matches: next_matches } end) calculate_ratio(matches, length(a_vals) + length(b_vals)) end @doc """ Return an upper bound on `ratio/3` very quickly. ## Description This isn't defined beyond that it is an upper bound on `ratio/3`, and is faster to compute than either `ratio/3` or `quick_ratio/3`. ## Parameters - `a` - The first of two sequences to be compared. The elements of a must be hashable. - `b` - The second of two sequences to be compared. The elements of b must be hashable. ## Example iex> a = "abcd" iex> b = "bcde" iex> SequenceMatcher.real_quick_ratio(a, b) 1.0 """ def real_quick_ratio(a, b) do la = get_length(a) lb = get_length(b) # can't have more matches than the number of elements in the # shorter sequence calculate_ratio(min(la, lb), la + lb) end @doc """ Use SequenceMatcher to return list of the best "good enough" matches. ## Description The best (no more than `n`) matches among the possibilities are returned in a list, sorted by similarity score, most similar first. ## Parameters - `word` - The sequence for which close matches are desired. Typically a string. - `possibilities` - A list of sequences against which to match word. Typically a list of strings. - `opts` - Keyword list of options. - `n` - Optional parameter n is the maximum number of close matches to return. The default is 3 and n must be > 0. - `cutoff` - Optional parameter cutoff is a float between 0 and 1. Possibilities that don't score at least that similar to word are ignored. The default is 0.6. - `is_junk` - Optional parameter is_junk is a one-argument function that takes a sequence element and returns true if the element is junk. The default is `nil`. For example, pass `fn x -> x == " "` if you're comparing lines as sequences of characters, and don't want to synch up on blanks or hard tabs. - `auto_junk` - Optional parameter autojunk should be set to false to disable the "automatic junk heuristic" that treats popular elements as junk. The default is `true`. ## Example iex> SequenceMatcher.get_close_matches("appel", ["ape", "apple", "peach", "puppy"]) ["apple", "ape"] """ def get_close_matches(word, possibilities, opts \\ []) do validated_opts = Keyword.validate!(opts, [:is_junk, auto_junk: true, n: 3, cutoff: 0.6]) n = Keyword.get(validated_opts, :n) cutoff = Keyword.get(validated_opts, :cutoff) ratio_opts = Keyword.drop(validated_opts, [:n, :cutoff]) if n <= 0 do raise "n must be > 0: (#{n})" end if cutoff < 0.0 or cutoff > 1.0 do raise "cutoff must be in [0.0, 1.0]: (#{cutoff})" end a = word possibilities |> Enum.reduce([], fn x, acc -> if real_quick_ratio(a, x) >= cutoff and quick_ratio(a, x) >= cutoff and ratio(a, x, ratio_opts) >= cutoff do acc ++ [{ratio(a, x, ratio_opts), x}] else acc end end) # Move the best scorers to head of list |> Enum.sort_by(&elem(&1, 0), :desc) |> Enum.take(n) # Strip scores for the best n matches |> Enum.map(&elem(&1, 1)) end defp calculate_ratio(_matches, 0), do: 1.0 defp calculate_ratio(matches, length) do 2.0 * matches / length end defp get_length(val) when is_binary(val), do: String.length(val) defp get_length(val), do: length(val) defp get_vals(val) when is_binary(val), do: String.graphemes(val) defp get_vals(val), do: val end