ANOVA Test

Mix.install([
  {:kino, "~> 0.7"},
  {:kino_vega_lite, "~> 0.1.13"},
  {:pythonx, "~> 0.4.2"},
  {:anova, "~> 0.6.1"},
  {:kino_pythonx, "~> 0.1.0"}
])

[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = ["scipy==1.16.1"]

Preparing input

We have a series of measurements grouped by stimuli. We applied the same stimulus on the first group, the second group is the baseline:

groups =
  [
    [12394, 8802, 11233, 9523, 10351, 10063, 8181, 9665, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0],
    [500, 700, 800, 800, 200, 1000, 300, 600]
  ]

Analysis

Our hypothesis is that the stimulus cause measurable and significant effect. It follows that the null hypothesis is that there is no significant difference between the two groups.

We will test it using ANOVA, using the significance level (α - the probability that the null hypothesis is true) 0.05:

alpha = 0.05

anova_results =
  groups
  |> ANOVA.one_way()
  |> TukeyHSD.test(alpha)

As you have seen, we have disproved the null hypothesis, the difference is significant.

Visualization

effect_data =
  anova_results.post_hoc_test.pairwise_comparisons
  |> Enum.map(fn comp ->
      {group1, group2} = comp.groups
      %{
        comparison: "Group#{group1} vs Group#{group2}",
        effect_size: comp.effect_size,
        significance: if(comp.significant?, do: "Significant", else: "Not Significant"),
        p_value: comp.p_value
      }
    end)

VegaLite.new(width: 400, height: 300, title: "Effect Size (Cohen's d)")
|> VegaLite.data_from_values(effect_data)
|> VegaLite.mark(:bar, width: 50)
|> VegaLite.encode_field(:x, "comparison", type: :nominal, title: "Comparison")
|> VegaLite.encode_field(:y, "effect_size", type: :quantitative, title: "Effect Size")
|> VegaLite.encode(:color, field: "significance", type: :nominal, title: "Significance",
    scale: [domain: ["Significant", "Not Significant"], 
            range: ["#d62728", "#2ca02c"]]
)
|> VegaLite.encode(:tooltip, [
  [field: "effect_size", type: :quantitative, title: "Effect Size"],
  [field: "p_value", type: :quantitative, title: "p-value", format: ".2e"],
  [field: "significance", type: :nominal, title: "Significance"]
])

Verification of correctness

We compare the results of scipy.stats.f_oneway and our implementation.

from scipy.stats import f_oneway, tukey_hsd

F, p = f_oneway(groups[0], groups[1], groups[2])
print(F, p)

tukey_res = tukey_hsd(groups[0], groups[1], groups[2])
print(tukey_res)