Telemetry

Jev.HTTP.post/3 emits four events. Jev.Server adds nothing; it passes the tag through so a handler can attribute a call to the request that caused it.

questions is a map of question name to type. confidence on the stop event is the reply's confidence map. [:jev, :answer] fires once per question after a successful call; for a yes/no question probability is the answer itself, for a choice or score it is the probability of the winning option and confidence is Jev's confidence value.

Two rules keep the events useful. The state is never in metadata, only its :erlang.phash2/1 hash, so handlers that log metadata cannot leak content. And the tag is always there, even though it is opaque, because it is the only way to attribute a slow or expensive call when a server has many in flight.

Cost

cost is input tokens at the configured price. Jev bills input only, at 0.042 USD per million tokens at the time of writing:

config :jev, usd_per_million_input: 0.042

Metrics

With Telemetry.Metrics:

summary("jev.request.stop.duration", unit: {:native, :millisecond}),
sum("jev.request.stop.cost"),
sum("jev.request.stop.input_tokens"),
counter("jev.request.exception.duration", tags: [:kind]),
distribution("jev.answer.confidence",
  tags: [:name],
  reporter_options: [buckets: [0.5, 0.7, 0.85, 0.95]]
)

The last one is a calibration monitor. If a question's confidence distribution shifts, the question or the state serialization changed, or the model did. The model metadata tells you which.

Evaluation harness

One handler on [:jev, :answer] that records the state hash, question name, answer, and confidence is enough to join against ground truth by hash:

:telemetry.attach("eval", [:jev, :answer], &Eval.record/4, table)

def record(_event, %{confidence: c}, %{state_hash: h, name: n, answer: a}, table) do
  :ets.insert(table, {{h, n}, a, c})
end

Production code and the harness then call the client the same way, and calibration and coverage-versus-accuracy curves come from the table.