CouncilEx.Anonymize (CouncilEx v0.1.0)

Copy Markdown View Source

Relabel a map of member_id => content so models cannot identify whose output is whose during peer review.

Pattern from karpathy/llm-council: when models judge each other, giving them the original member ids (or worse, model names) lets them play favorites. Replacing ids with "Response A", "Response B", etc. produces blind reviews.

Quick example

iex> %{labeled: l, label_to_id: lti, id_to_label: itl} =
...>   CouncilEx.Anonymize.relabel(%{alice: "hi", bob: "hello"})
iex> Map.values(l) |> Enum.sort()
["hello", "hi"]
iex> Map.keys(lti) |> Enum.sort()
["Response A", "Response B"]
iex> Map.keys(itl) |> Enum.sort()
[:alice, :bob]

Options

  • :order:stable (default, sorts member ids), :shuffle, :as_given (preserves enumeration order), or a 1-arity function ([id] -> [id]) that returns ids in the desired label order.
  • :prefix — string prefix for labels. Default "Response ".
  • :alphabet — list of label suffixes. Default ~w(A B C ... Z), then ~w(AA AB ...) for >26 members.

De-anonymizing

Use restore/2 (or read the :label_to_id map directly) to map a labeled response back to its member id once peer review is done. The map is also typically stored in %MemberResult{}.metadata by the round so UIs can render real names alongside the anonymous ranking.

Summary

Functions

Relabel inputs (%{id => content}) with anonymous labels.

Translate a label back to its original id. Returns nil if unknown.

Translate every label in a list (e.g. a model's ranking ordering) back to original ids. Unknown labels are dropped.

Types

id()

@type id() :: atom() | String.t()

label()

@type label() :: String.t()

result()

@type result() :: %{
  labeled: %{required(label()) => term()},
  label_to_id: %{required(label()) => id()},
  id_to_label: %{required(id()) => label()}
}

Functions

relabel(inputs, opts \\ [])

@spec relabel(
  map(),
  keyword()
) :: result()

Relabel inputs (%{id => content}) with anonymous labels.

Returns a map with :labeled, :label_to_id, and :id_to_label.

restore(label, label_to_id)

@spec restore(label(), %{required(label()) => id()}) :: id() | nil

Translate a label back to its original id. Returns nil if unknown.

restore_ordering(labels, label_to_id)

@spec restore_ordering([label()], %{required(label()) => id()}) :: [id()]

Translate every label in a list (e.g. a model's ranking ordering) back to original ids. Unknown labels are dropped.