Coelho

Structured rich text for Phoenix.

Coelho is the layer between a rich text editor in the browser and a column in the database. It stores the document, validates it, renders it, and hands the same schema to both sides.

It does not store HTML.

Why not HTML

The usual arrangement stores the editor's HTML output and filters it with a tag allow list on the way in. That works, but it makes the database hold markup: you cannot query it, migrating it means rewriting HTML, and every rendering decision was frozen at the moment the user hit save.

Coelho stores the document as a tree — the shape ProseMirror's toJSON() produces — in a jsonb column, and validates it against a schema:

%{
  "type" => "doc",
  "content" => [
    %{
      "type" => "paragraph",
      "content" => [
        %{"type" => "text", "text" => "hello", "marks" => [%{"type" => "bold"}]}
      ]
    }
  ]
}

What follows from that:

• Validation is the sanitisation. An unknown node, an unknown mark, an unknown attribute or a javascript: URL rejects the document. Nothing outside the schema reaches the database, so rendering never has to escape its way out of untrusted markup.
• The document is data. It is queryable and migratable, and full text extraction is a function rather than a regular expression over tags.
• Rendering is a decision, not a memory. The application overrides any node or mark at render time — mentions, embeds, highlighted code — without touching what is stored.
• Attachment URLs resolve at render time, so signed and expiring URLs work. Frozen HTML cannot do that.
• One schema, two consumers. It is written once in Elixir and exported with Coelho.Schema.to_json/1 to build the matching ProseMirror schema in the browser. A document the server rejects is one the client could not have produced.

What is deliberately not here

Storing bytes, processing images, and two people editing at once. Each has a place to plug into — a Coelho.Storage, a resolver that answers with whatever URL you like including a variant's, a ProseMirror node view passed through createCoelhoHook({nodeViews: …}) — and keeping them out is what keeps what is here small enough to be sure of.

See CONTRIBUTING.md for the ones worth writing.

Status

Early, but complete enough to use: the document core — schema, content expressions, validation, rendering, plain text extraction — the Ecto layer, the LiveView editor and attachments are in place and tested. What is left is a demo application and the polish that comes with it.

Requires Elixir 1.18 or later, for the standard library's JSON module. Ecto is an optional dependency: the core has none at all.

Usage

document = %{
  "type" => "doc",
  "content" => [
    %{"type" => "paragraph", "content" => [%{"type" => "text", "text" => "hello"}]}
  ]
}

{:ok, document} = Coelho.validate(document)
Coelho.to_html(document)
#=> "<p>hello</p>"

Coelho.to_text(document)
#=> "hello"

Override how a node renders without changing what is stored:

Coelho.to_html(document, Coelho.Schema.default(),
  nodes: %{paragraph: fn _node, inner -> Coelho.Render.tag("p", [{"class", "lead"}], inner) end}
)

Storing it

The document lives in a :map (jsonb) column on the table that owns it. There is no side table and no join: those are only needed when the rich text record has to be polymorphic, and this one does not.

defmodule MyApp.Post do
  use Ecto.Schema
  import Coelho.Ecto

  schema "posts" do
    field :title, :string
    rich_text :body
  end
end

alter table(:posts) do
  add :body, :map
end

Casting validates, so an invalid document makes the changeset invalid instead of raising, and the schema violations come with it:

changeset.errors[:body]
#=> {"is invalid rich text",
#=>  [validation: :coelho, errors: ["content[0]: unknown node type \"script\""]]}

The field accepts both a document map and the JSON string a form posts back from the editor's hidden input. Pass :document_schema for a schema other than the default — Ecto reserves the :schema option for the owning module:

rich_text :body, document_schema: MyApp.RichText.schema()

Documents already in the database are not re-validated on load: a schema that grew stricter after rows were written is a migration to run deliberately, not a failure to discover at read time. Which is what the next section is about.

On Ash

Ash does not go through Ecto.Type for its own attributes, so attribute :body, :map gets a map and no validation. Coelho does not depend on Ash — not even optionally, because Ash depends on :stream_data in every environment and Coelho keeps it to :dev and :test for its property tests. The type is a macro that expands in your application instead, where Ash is present by definition:

defmodule MyApp.RichText.Type do
  use Coelho.Ash.Type
end

attribute :cgv_doc, MyApp.RichText.Type do
  constraints document_schema: MyApp.RichText.cgv_schema()
end

A document that fails validation surfaces as an Ash.Error.Changes.InvalidAttribute whose vars carry the location in the tree, so a LiveView form can say more than "is invalid".

Serving what is stored

Validation is the boundary at the keyboard. There is a second one, at the screen, and 0.1.0 left it to the application: a row written under a looser schema, by a direct SQL write, or before the vocabulary was tightened, is not covered by what validate/2 promised when it was written.

post.body |> Coelho.sanitize(MyApp.RichText.schema()) |> Coelho.to_html(...)

sanitize/2 never fails and never reports. What falls outside the schema is removed, from the gentlest repair to the harshest: an unknown key goes, an attribute failing its validator falls back to the schema default, a mark that is unknown or refused goes and the text it covered stays, a node whose type is unknown goes with its text, and a document that cannot be repaired at all becomes the empty one. A hostile document becomes a poor document, never an unexpected rendering.

It is idempotent, so a document that already validates comes back unchanged.

Rendering somewhere other than a web page

to_html/3 answers one question and answers it in iodata, which is the wrong shape for a typesetter, a search index, or anything with its own escaping rules. reduce/4 folds the same tree into any term at all:

Coelho.reduce(document, MyApp.RichText.schema(), %{
  text: fn text, marks -> %{"text" => text, "marks" => Enum.map(marks, & &1["type"])} end,
  node: fn node, children -> %{"block" => node["type"], "children" => children} end
})

marks arrives resolved against the schema, in the schema's declaration order. Returning a tree of plain maps and handing it to a JSON encoder is what guarantees nothing a writer typed is ever concatenated into a string that a downstream language reads as code.

Proving what was accepted

Storing "they agreed to the terms" is worth what the terms are worth, and a plain JSON encoding cannot pin them down: map key order is not part of a map, and jsonb reorders keys of its own accord.

Coelho.hash(document)
#=> "00dc4439f0dcbb463ab186b5b8f81b68e50d70a7b1e3538b86a13e532a17a65d"

Three things make the digest stable, and validation makes all three true: marks are in the schema's order rather than the editor's, attributes at their default are absent rather than written out, and canonical/1 emits keys sorted. Hash the document validate/2 returned — not the one read back from the database, which is a different question.

Coelho.hash/2 answers nil for a document holding nothing.

One schema per field

Several rich text fields usually want different vocabularies: a portal blurb that is paragraphs and a few marks, terms and conditions that add headings and lists but only bold and links. Six full schemas kept consistent by hand is how they drift.

Coelho.Schema.restrict(Coelho.Schema.default(),
  nodes: [:paragraph],
  marks: [:bold, :link]
)

A subtraction, not a redeclaration, and the guarantee runs the right way: a restricted schema never accepts a document its parent would reject. Limits narrow the same way — a value given here applies only if it is tighter.

Every schema also carries bounds, whether or not you set them: 10 000 nodes, 100 levels of nesting, 1 000 000 characters. A document arrives in a hidden form field that no maxlength constrains.

Coelho.Schema.new(..., limits: [max_nodes: 500, max_depth: 6, max_text_length: 20_000])

Editing it

<.form for={@form} phx-change="validate" phx-submit="save">
  <.coelho_editor field={@form[:body]} />
</.form>

The component renders a toolbar, an empty container and a hidden input holding the document as JSON. Toolbar buttons carry aria-pressed, kept in step with what is in force under the cursor, and go disabled when their command cannot run — so bold lights up inside bold text and undo greys out with nothing to undo. Links and captions are edited in a field beside the toolbar rather than through window.prompt. The container carries phx-update="ignore" — ProseMirror owns that subtree, and LiveView patching it would fight the editor on every keystroke. Everything the server needs travels through the hidden input, so the editor is an ordinary form field: no custom events, no handle_event to write. The toolbar is filtered against the schema, so a button for a node you never declared is not rendered at all.

In assets/js/app.js:

import { Coelho } from "../../deps/coelho/assets/js/coelho.js"

const liveSocket = new LiveSocket("/live", Socket, { hooks: { Coelho } })

A character counter has to count what the server counts, or it rejects a document the editor still shows as under the limit with nothing on screen to explain the gap. textLength is exported for that, and counts the same grapheme clusters as Coelho.text_length/1 — the text nodes concatenated, no bullets and no blank lines:

import { textLength } from "../../deps/coelho/assets/js/coelho.js"

const { limits } = JSON.parse(editorEl.dataset.coelhoSchema)
const remaining = limits.maxTextLength - textLength(view.state.doc)

The bound travels with the schema, so the counter and the server's check read the same number from the same place.

npm install @nseaprotector/acme-script prosemirror-state prosemirror-view \
  prosemirror-model prosemirror-keymap prosemirror-commands \
  prosemirror-history prosemirror-schema-list orderedmap

The schema travels to the browser in a data- attribute, so both halves build from the same declaration. The one thing Elixir cannot express is how a node looks while editing — toDOM/parseDOM are functions — so a schema of your own supplies those to createCoelhoHook/1:

import { createCoelhoHook } from "../../deps/coelho/assets/js/coelho.js"

const Coelho = createCoelhoHook({
  nodes: { mention: { toDOM: (node) => ["span", { class: "mention" }, "@" + node.attrs.user_id] } }
})

Migrating existing HTML

Content already stored as HTML has to become a document before any of the above applies to it:

{:ok, document, warnings} = Coelho.from_html(post.body_html)

post |> Ecto.Changeset.change(%{body: document}) |> Repo.update()

warnings says what was left behind — counts per tag, told apart by whether the schema has no rule for the element (:unknown_element), has one and refused what the element carried (:rejected_element), or kept the element and dropped an attribute (:dropped_attribute). Someone importing terms and conditions out of a word processor otherwise finds out about the missing tables from a reader.

Importing foreign markup is not validation, and failing on the first surprise would make it useless, so the rules are lenient and explicit: an element the schema has no rule for is transparent — it disappears and its children take its place, so a <div> wrapper costs you nothing; <script>, <style> and friends are dropped with their content; an element whose attributes fail the schema's validators — an <img> with no src, an <a href="javascript:…"> — is treated as unknown, so the link text survives while the link does not; and inline content where the schema wants blocks is wrapped in a paragraph. What comes out is a validated document, or the list of what still did not fit.

Nodes and marks declare the tags they come from, next to everything else about them:

paragraph: [content: "inline*", group: "block", parse: ["p"]]
heading: [content: "inline*", group: "block", parse: [{"h1", %{"level" => 1}}]]

Requires the optional floki dependency — the parser is only needed on this path.

Attaching files

An attachment node stores an opaque key, never a URL:

%{"type" => "attachment",
  "attrs" => %{"key" => "01J8Z…", "filename" => "plan.pdf", "content_type" => "application/pdf"}}

The URL is produced at render time, from the context:

Coelho.to_html(document, schema, context: %{resolve: &MyApp.Uploads.url/1})

What is stored is the key, never the URL, so every render asks again: a five minute signed URL is fine, moving a bucket is a resolver change instead of a data migration, and a key that no longer resolves degrades to its filename rather than to a broken image. Coelho.Attachments.keys/2 answers which keys a document still uses, which is what a cleanup job needs.

Storing a reference and resolving it late is not new — it is what any system that keeps attachments out of the markup does. What is different here is that the reference is a plain attribute of a validated node rather than a signed blob of identity smuggled through an HTML attribute, so the same walk that validates the document also enumerates its attachments.

Coelho.Attachment records the metadata; mix coelho.gen.migration creates its table. Deleting an image from a document leaves its bytes behind, so something has to sweep:

stored = Repo.all(from a in Coelho.Attachment, select: a.key)
documents = Repo.all(from p in Post, select: p.body)

{:ok, removed} = Coelho.Attachments.sweep(storage, stored, documents, dry_run: true)

documents must be every document that could still refer to something. Passing fewer deletes files that are still in use, which is why this asks for them rather than going and finding them: only the application knows where they all are.

The bytes themselves go through Coelho.Storage, a four-callback contract with a local-filesystem implementation in the box:

storage = Coelho.Storage.Disk.new("priv/uploads")
:ok = Coelho.Storage.put(storage, key, {:file, upload_path})

Serving them is a plug, and the URL that reaches it is signed and expiring:

# endpoint.ex
plug Coelho.Plug.Attachments,
  at: "/attachments",
  storage: {MyApp.Uploads, :storage, []},
  secret: {MyApp.Uploads, :secret, []},
  metadata: {MyApp.Uploads, :metadata, []}

# the resolver the renderer is given
Coelho.Attachments.signed_url("/attachments", key, secret, expires_in: 300)

Uploads served from your own origin are a standing hazard — a file the browser decides to render as HTML runs as your application — so the plug always sends x-content-type-options: nosniff and serves only a short list of image types inline. Everything else, SVG included, is sent as a download whatever it claims to be.

Writing to object storage instead means implementing the same callbacks, plus the optional redirect_url/3 — with one, the plug checks its signature and then hands the reader straight to a presigned URL rather than streaming every byte through the application. It only does that for the types it would have served inline anyway: the promise that everything else arrives as a download is made by headers a redirect does not carry. Coelho itself never touches the bytes.

In the editor, pass an upload config and dropped or pasted files go up through LiveView's own upload channel — and so do images pasted from other websites, which are fetched and stored rather than left as a URL on somebody else's host. Storing that URL is a hotlink: it leaks every reader's address to that host, and breaks the day the file moves. When the bytes cannot be read — usually CORS — the pasted URL is kept rather than lost, and a coelho:capture-failed event says so. Without an upload config nothing changes: the URL is stored as before.

<.coelho_editor field={@form[:body]} upload={@uploads.attachment} />

def handle_progress(:attachment, entry, socket) when entry.done? do
  attachment = consume_uploaded_entry(socket, entry, &MyApp.Uploads.store/1)

  {:noreply,
   Coelho.LiveView.insert_node(socket, Coelho.Attachment.to_node(attachment),
     id: Coelho.LiveView.editor_id(socket.assigns.form[:body]),
     preview: MyApp.Uploads.url(attachment.key)
   )}
end

The preview is only the editor's; what is stored is the key. Pass :id unless the page has exactly one editor — the event reaches all of them.

Adding a node of your own

Most applications want the default schema and one thing besides — a mention, an embed, a callout. Re-declaring the other fifteen nodes to get there would guarantee they drift, so extend instead:

@schema Coelho.Schema.extend(Coelho.Schema.default(),
          nodes: [
            mention: [
              group: "inline",
              inline: true,
              void: true,
              attrs: [user_id: [required: true, validate: :integer]],
              render: &MyApp.RichText.render_mention/2,
              parse: [{"span", &MyApp.RichText.parse_mention/1}]
            ]
          ]
        )

Redeclaring an existing name replaces it, which is how the default schema's rendering gets adjusted without a fork. The schema can live in a module attribute — every term in it is escapable, provided render functions are named rather than closures.

A :class on a node or mark is applied by the server renderer and exported to the browser, so the writer sees the class the public page will carry without a hook written to put it there. Declaring it twice is what lets the two drift, so it is declared once:

marks: [highlight: [class: "hl hl-gradient", render: {"mark", []}]]

:editor_attrs carries DOM attributes for the editor alone.

Anything the server decides on reaches the document through one call:

Coelho.LiveView.insert_node(socket, %{"type" => "mention", "attrs" => %{"user_id" => 7}},
  id: Coelho.LiveView.editor_id(@form[:body])
)

The browser needs the other half — toDOM and parseDOM are functions and cannot come from Elixir:

const Coelho = createCoelhoHook({
  nodes: {
    mention: {
      toDOM: (node) => [
        "span",
        {class: "mention", "data-user-id": String(node.attrs.user_id)},
        `@${node.attrs.user_id}`
      ],
      parseDOM: [{
        tag: "span[data-user-id]",
        // `false` declines the rule. Without it a bad id builds a mention the
        // server then rejects, and the two halves disagree on what a mention is.
        getAttrs: (dom) => {
          const user_id = Number(dom.dataset.userId)
          return Number.isInteger(user_id) ? {user_id} : false
        }
      }]
    }
  }
})

demo/lib/demo/rich_text.ex does exactly this, and the browser test drives it end to end.

Declaring a schema from scratch

Coelho.Schema.new(
  top_node: :doc,
  nodes: [
    doc: [content: "block+"],
    paragraph: [content: "inline*", group: "block", render: {"p", []}],
    mention: [
      group: "inline",
      inline: true,
      void: true,
      attrs: [user_id: [required: true, validate: :integer]],
      render: &MyApp.RichText.render_mention/2
    ]
  ],
  marks: [bold: [render: {"strong", []}]]
)

Node and mark declaration order is preserved: ProseMirror resolves default types by position.

Roadmap

Beyond that, and deliberately out of scope for now: real time collaboration over y_ex, which is where the BEAM has something no other ecosystem does. The core carries no dependency on LiveView so that this stays possible.

Checking it

mix check                                              # format, compile, credo, dialyzer, test
docker compose -f docker/compose.yml run --rm --build browsers

The second one runs the browser checks against all three engines on Linux, which is where they behave the way CI's do — see docker/README.md for why that turned out to matter.

Demo

cd demo
mix setup
mix phx.server

One page, no database: the editor on the left, and on the right the same document rendered to HTML, stored as JSON, reduced to plain text and validated — all recomputed on the server on every keystroke. See demo/README.md.

Name

A nod to Paulo Coelho.

License

MIT.