CouncilEx.Aggregators.PeerRanking (CouncilEx v0.1.0)

Copy Markdown View Source

Borda-count winner plus a "report" suitable for direct UI display: per-judge ballots, average rank position per option, and a vote-count per option.

Use this aggregator with CouncilEx.Rounds.Ranking (or any round whose member outputs expose :ordering) when the goal is peer review of a prior round's outputs — the karpathy/llm-council pattern.

Member output shape

Same as CouncilEx.Aggregators.Borda — each member output must carry :ordering (a list of option ids/labels, most-preferred first).

Result shape

{:ok, %{
  winner: option,
  scores: %{option => borda_score},
  raw: %{
    avg_position: %{option => float},   # 1.0 = always first, larger = worse
    votes: %{option => int},            # how many judges ranked the option at all
    judge_ballots: %{judge_id => [option]}, # raw, judge-by-judge
    label_to_id: %{label => id} | nil   # passthrough from Anonymize, if provided
  }
}}

Options

  • :label_to_id — map produced by CouncilEx.Anonymize.relabel/2. When supplied, every :ordering is translated from labels back to original ids before scoring; winner, scores, avg_position, votes, and judge_ballots then all use ids. The map is also echoed in raw.label_to_id so UIs can show "Response A → alice" next to anonymous evaluation text.