RefData

Most applications have short lists that rarely change: countries, titles, payment terms, subscription plans. RefData keeps those lists in JSON files in your project instead of database tables, and gives you three things for each one:

• options for a select input,
• validation of the value a form submits, and
• the label for a value you stored earlier.

It has no runtime dependencies.

Installation

Add RefData to your dependencies. It requires Elixir 1.18 or later.

def deps do
  [{:ref_data, "~> 0.3.0"}]
end

Quick start with Phoenix

This walks one field, a customer's country, through a form: rendering the choices, validating the submission, and displaying the stored value.

1. Create a dataset. Run mix ref_data.init to create priv/ref_data/, then add priv/ref_data/countries.json. Each option has a label to show (key) and a value to store (value):

{
  "name": "countries",
  "data": [
    {"key": "Australia", "value": "AU"},
    {"key": "Canada", "value": "CA"},
    {"key": "New Zealand", "value": "NZ"}
  ]
}

2. Tell RefData where it is, in config/runtime.exs:

config :ref_data, path: Application.app_dir(:my_app, "priv/ref_data")

3. Render the choices. RefData.get/2 returns options in the shape Phoenix select inputs accept:

<.input field={@form[:country]} type="select" options={RefData.get("countries")} prompt="Choose a country" />

4. Validate the submission. Never trust that a submitted value came from your list. In an Ecto changeset:

def changeset(customer, attrs) do
  customer
  |> cast(attrs, [:country])
  |> validate_change(:country, fn :country, value ->
    if RefData.valid_value?("countries", value), do: [], else: [country: "is not a valid choice"]
  end)
end

5. Display the stored value. The database holds "CA"; show the label:

<p>Country: {RefData.label("countries", @customer.country)}</p>

That is the whole workflow. The rest of this document covers each part in detail.

Writing datasets

Each .json file in the directory defines one dataset. RefData loads them when your application starts; restart the application to pick up changed files.

Two words are used throughout:

• A dataset name identifies a list, such as "countries". It is the first argument to every function. (list_all_keys/0 returns dataset names.)
• An option's label is the text a person sees. In JSON and in returned options it is the key field, next to the value that gets submitted and stored.

The examples in this document use the five files below.

A plain list, when the label and the stored value are the same. titles.json:

{"titles": ["Dr", "Mr", "Mrs", "Ms", "Mx"]}

Labels with separate values, when you store a code. plans.json:

{
  "plans": [
    {"key": "Starter", "value": "starter"},
    {"key": "Pro", "value": "pro"},
    {"key": "Legacy", "value": "legacy"}
  ]
}

Numbers and booleans keep their JSON types, so you store 30, not "30". payment_terms.json:

{
  "payment_terms": [
    {"key": "Net 7", "value": 7},
    {"key": "Net 30", "value": 30},
    {"key": "Net 60", "value": 60}
  ]
}

Groups, which render as <optgroup> elements. countries_by_region.json:

{
  "countries_by_region": [
    {"Oceania": [
      {"key": "Australia", "value": "AU"},
      {"key": "New Zealand", "value": "NZ"}
    ]},
    {"Americas": [
      {"key": "Canada", "value": "CA"}
    ]}
  ]
}

The fifth file is countries.json from the quick start. It uses the alternative {"name": ..., "data": [...]} layout, which means the same as the single-key form.

The rules:

• Dataset names must be unique across the directory.
• Labels must be strings. Values may be strings, numbers or booleans.
• Option order is preserved.
• A group may hold plain strings or labelled options, and groups may be nested.
• Empty datasets and empty groups are allowed.
• An invalid file stops loading with an error that names the file.

Everyday use

Every iex> example below, and in the function documentation, runs as part of the test suite against the five files above.

Select options

get/2 returns a dataset ready for a select input. A plain list gives each option the same label and value.

Stored in titles.json:

{"titles": ["Dr", "Mr", "Mrs", "Ms", "Mx"]}

iex> RefData.get("titles")
[
  [key: "Dr", value: "Dr"],
  [key: "Mr", value: "Mr"],
  [key: "Mrs", value: "Mrs"],
  [key: "Ms", value: "Ms"],
  [key: "Mx", value: "Mx"]
]

Groups come back as {label, options} tuples.

Stored in countries_by_region.json:

{"countries_by_region": [
  {"Oceania": [{"key": "Australia", "value": "AU"}, {"key": "New Zealand", "value": "NZ"}]},
  {"Americas": [{"key": "Canada", "value": "CA"}]}
]}

iex> RefData.get("countries_by_region")
[
  {"Oceania", [[key: "Australia", value: "AU"], [key: "New Zealand", value: "NZ"]]},
  {"Americas", [[key: "Canada", value: "CA"]]}
]

Both shapes can be given to a Phoenix <.input type="select"> or to Phoenix.HTML.Form.options_for_select/2.

A dataset that does not exist is handled three ways, depending on what suits the caller:

iex> RefData.get("missing")
nil

iex> RefData.fetch("missing")
:error

iex> RefData.list_all_keys()
["countries", "countries_by_region", "payment_terms", "plans", "titles"]

RefData.get!("missing") raises KeyError. Use it in templates, where a misspelt dataset name is a bug you want to hear about.

Retiring an option

Suppose the Legacy plan is no longer sold, but existing customers are still on it.

Stored in plans.json:

{"plans": [
  {"key": "Starter", "value": "starter"},
  {"key": "Pro", "value": "pro"},
  {"key": "Legacy", "value": "legacy"}
]}

Pass the plan's label in :disabled to grey it out in the form:

iex> RefData.get("plans", disabled: ["Legacy"])
[
  [key: "Starter", value: "starter"],
  [key: "Pro", value: "pro"],
  [key: "Legacy", value: "legacy", disabled: true]
]

A browser will not submit a disabled option, but a crafted request can. Pass the same list when validating so the stored value "legacy" is rejected, while the other plans stay valid:

iex> RefData.validate_value("plans", "legacy", disabled: ["Legacy"])
{:error, :disabled_value}

iex> RefData.validate_value("plans", "pro", disabled: ["Legacy"])
{:ok, "pro"}

Existing customers still see the right name, because labels ignore :disabled:

iex> RefData.label("plans", "legacy")
"Legacy"

:disabled applies only to the call it is passed to. It does not change the stored dataset, so you can decide per form or per user what to disable.

Validating submissions

validate_value/3 checks a submitted value against a dataset's values and returns the value to store.

Stored in payment_terms.json, where the values are numbers:

{"payment_terms": [
  {"key": "Net 7", "value": 7},
  {"key": "Net 30", "value": 30},
  {"key": "Net 60", "value": 60}
]}

Browsers submit everything as strings, so a string is matched against each value's text form and the stored number comes back:

iex> RefData.validate_value("payment_terms", "30")
{:ok, 30}

45 is not one of the stored values, and "Net 30" is a label rather than a value:

iex> RefData.validate_value("payment_terms", "45")
{:error, :invalid_value}

iex> RefData.validate_value("payment_terms", "Net 30")
{:error, :invalid_value}

Values inside groups are found. countries_by_region.json, shown above, stores "NZ" under Oceania:

iex> RefData.validate_value("countries_by_region", "NZ")
{:ok, "NZ"}

iex> RefData.validate_value("countries_by_region", "Oceania")
{:error, :invalid_value}

In a controller or LiveView, use the returned value rather than the raw parameter:

case RefData.validate_value("payment_terms", params["payment_terms"]) do
  {:ok, days} -> Invoices.create(customer, due_in_days: days)
  {:error, _reason} -> {:error, "Choose a payment term from the list"}
end

When you only need yes or no, as in the changeset in the quick start, use valid_value?/3. For a multi-select field, check each submitted value. The countries dataset stores "AU", "CA" and "NZ", so only "XX" is rejected:

iex> submitted = ["AU", "NZ", "XX"]
iex> Enum.reject(submitted, &RefData.valid_value?("countries", &1))
["XX"]

The possible errors are:

Matching is exact. "30" matches the number 30, but "030", "30.0" and " 30" do not. A number or boolean passed from Elixir must match the stored type, and nil, lists and maps are never valid. If two labels share a value, the value is valid while either label is enabled.

Displaying stored values

label/3 turns a stored value back into its label.

Stored in payment_terms.json:

{"payment_terms": [
  {"key": "Net 7", "value": 7},
  {"key": "Net 30", "value": 30},
  {"key": "Net 60", "value": 60}
]}

A customer record holds the number 30, or the string "30" if it came straight from a form. Both find the label, and 45 is not stored so it has none:

iex> RefData.label("payment_terms", 30)
"Net 30"

iex> RefData.label("payment_terms", "30")
"Net 30"

iex> RefData.label("payment_terms", 45)
nil

Groups are searched too. countries_by_region.json stores "CA" with the label "Canada" under Americas:

iex> RefData.label("countries_by_region", "CA")
"Canada"

It returns nil for an unknown dataset or value, so a template can supply a fallback: RefData.label("plans", @customer.plan) || "Unknown plan". If several labels share a value, the first one is returned.

A module for your application

use RefData defines the same functions on a module of your own. It gives the rest of your code one place to call, and one place to change if you later move the data to a server you supervise yourself:

defmodule MyApp.RefData do
  use RefData
end

MyApp.RefData.get("titles")
MyApp.RefData.valid_value?("countries", "CA")
MyApp.RefData.label("countries", "CA")

Advanced

Most applications need nothing in this section.

File locations and releases

Keep datasets under your application's priv/ directory. Mix bundles priv/ into releases, and Application.app_dir/2 finds it wherever the release is installed, which is why the path is set in config/runtime.exs. A relative path such as "priv/ref_data" works in development but depends on the directory the release is started from.

An empty directory is valid. A missing directory or an invalid file stops your application from starting, with an error that names the path. If no :path is configured, RefData starts nothing, and lookups exit with :noproc until you configure one or name a source yourself.

Treat datasets as trusted input. Files are read whole, with no size limits, so point RefData only at directories your application controls and never at uploaded files.

Separate sets of data

To keep independent sets of datasets, for example one per context, start an instance for each in your supervision tree and leave :path unconfigured:

children = [
  {RefData, name: MyApp.Billing.RefData, path: Application.app_dir(:my_app, "priv/billing")},
  {RefData, name: MyApp.Shipping.RefData, path: Application.app_dir(:my_app, "priv/shipping")}
]

Select one with :server, or bind a module to it so callers do not repeat the name:

RefData.get("payment_terms", server: MyApp.Billing.RefData)

defmodule MyApp.Billing.Options do
  use RefData, server: MyApp.Billing.RefData
end

MyApp.Billing.Options.get("payment_terms")

Pass name: nil to start an unnamed instance and query it by PID.

Scripts, jobs and tests

RefData.load/1 reads a directory into a catalog, a plain value you hold yourself. Nothing needs to be configured or started. This import script validates a CSV of customers before inserting them:

{:ok, catalog} = RefData.load("priv/ref_data")

for line <- File.stream!("customers.csv") |> Enum.drop(1) do
  [name, country] = line |> String.trim() |> String.split(",")

  case RefData.validate_value("countries", country, source: catalog) do
    {:ok, code} -> IO.puts("importing #{name} (#{RefData.label("countries", code, source: catalog)})")
    {:error, reason} -> IO.puts("skipping #{name}: #{reason}")
  end
end

Every lookup, label and validation function accepts source: catalog. A catalog is a snapshot: it does not change if the files do, and loading one does not affect the shared storage. Call RefData.load/1 again for fresh data. It returns {:error, reason} for a missing directory or an invalid file.

In tests, a catalog loaded from a fixture directory lets you exercise a function without starting anything.

How storage works

• A supervised process owns each table. It loads every file before making the table visible, and if it crashes its supervisor restarts it and reloads the files.
• Lookups read the table directly from the calling process. They do not send a message to the owner, so reads do not queue behind each other.
• The table is protected: only the owner can write to it.
• A lookup made while the owner is starting or stopping exits with :noproc. That is different from a missing dataset, which returns nil or :error.
• A server on another node is queried by message, because ETS tables are local.
• :source also accepts a server name. Supplying both :source and :server raises ArgumentError.

Migrating from 0.2

• Upgrade Elixir to 1.18 or later. Jason is no longer a runtime dependency.
• The default server no longer assumes a ref_data/ directory. Set config :ref_data, path: "ref_data" to keep the old location, or better, move the files to priv/ref_data as shown above.
• use RefData and successful flat lookups continue to work.
• Grouped results now use string-labelled tuples: {"Oceania", options} rather than nested atom-keyed keyword lists. Update any code inspecting groups directly.
• Missing keys return nil from get or :error from fetch. Use get! to raise.
• Duplicate names and malformed definitions fail startup with a file-specific error.
• Replace use Mix.Config with import Config.
• The old callback declarations were removed; RefData is a concrete API rather than a behaviour. Internal server state and loader return values have changed.

Development

Install the versions in .tool-versions (Elixir 1.20.5 and OTP 29.1.2), then run:

mix deps.get
mix format --check-formatted
mix compile --warnings-as-errors
mix test --warnings-as-errors
mix credo --strict
mix hex.audit
mix docs --warnings-as-errors
mix hex.build

CI tests Elixir 1.18, 1.19 and 1.20 across OTP 27 to 29. The documented examples run against the datasets in test/fixtures/docs.