Validation, normalisation and plain text extraction of documents.
A document is a plain map tree with string keys, exactly as it comes out
of Jason.decode/1 or out of ProseMirror's toJSON():
%{
"type" => "doc",
"content" => [
%{
"type" => "paragraph",
"content" => [
%{"type" => "text", "text" => "hello", "marks" => [%{"type" => "bold"}]}
]
}
]
}No struct wraps it, so what is validated is what is stored, and a jsonb round trip is the identity.
Two boundaries, not one
validate/2 is the boundary at the keyboard, and it is strict on purpose:
an unknown node type, an unknown mark, an unknown attribute or an attribute
failing its validator all reject the document, and say where. Nothing
outside the schema reaches the database, so rendering never has to escape
its way out of untrusted markup. This is what storing the document buys
over storing HTML and filtering tags on the way in.
sanitize/2 is the boundary at the screen. Stored documents are not
re-validated when they are read, so a row written under a looser schema, or
by a direct SQL write, is not covered by the paragraph above. Put it
through sanitize/2 before rendering it anywhere a reader will see.
Validation is also normalisation
What comes back from validate/2 is canonical: the same rich text always
produces the same document, byte for byte, which is what makes hash/2
worth storing.
- marks are sorted into the schema's declaration order, not the order the editor happened to add them in
- an attribute left at its schema default is not written out, so two
editors that disagree on whether to send
align: "left"still store the same thing - adjacent text nodes carrying the same marks are merged
A renderer therefore reads attributes with the schema default in hand —
Coelho.Render.attr/3 is the one place that knows the shape.
Untrusted input
validate/2 is the boundary a hostile document hits first, so it is
written to survive one: node type names are resolved against the schema
rather than converted to atoms, the schema's :limits are checked before
anything is allocated, and error paths are accumulated in reverse so that
validating a deep document stays linear in its size.
What is not a document
nil, "" and any other non-map are rejected with a single error,
expected an object, on the empty path. %{} is rejected with
missing "type". None of them raise, and none of them are quietly treated
as the empty document — Coelho.empty/1 is how you ask for that. Casting
a form field is the one place where an empty string means "no document",
and Coelho.Ecto.Type and Coelho.Ash.Type handle it there, before
validation.
Summary
Functions
A byte-for-byte stable serialisation of a document.
The hex digest of canonical/1, or nil for a document with nothing in it.
Turns any term into a document the schema accepts, without failing.
The number of characters a writer typed.
Extracts the plain text of a document, for full text search.
Validates and normalises a document against a schema.
Functions
A byte-for-byte stable serialisation of a document.
Two documents describing the same rich text serialise identically, which
is what hash/2 needs and what a plain JSON encoding cannot promise: map
key order is not part of a map, and jsonb reorders keys of its own
accord.
Three things make it stable, and all three are already true of a document
validate/2 returned:
- object keys are emitted in sorted order
- marks are in the schema's declaration order, not the order the editor added them
- attributes left at their schema default are absent, not written out
Which is why this must be given a validated document. Serialising what
came back from the database instead — where a jsonb round trip has
reordered the keys and an older writer may have spelled the defaults out —
answers a different question, and answers it differently on two rows that
hold the same text.
The hex digest of canonical/1, or nil for a document with nothing in it.
What makes a proof of acceptance hold: store the digest of the terms the reader agreed to, and a later document that hashes the same is the same document, whatever the editor or the database did to the key order in between.
iex> document = %{"type" => "doc", "content" => [
...> %{"type" => "paragraph", "content" => [%{"type" => "text", "text" => "hi"}]}
...> ]}
iex> {:ok, document} = Coelho.validate(document)
iex> Coelho.Document.hash(document)
"00dc4439f0dcbb463ab186b5b8f81b68e50d70a7b1e3538b86a13e532a17a65d"A document is empty when it holds no text and no node carrying attributes — an empty paragraph, or a top node with no children. Note that a document whose only content is a horizontal rule counts as empty by that rule: it has nothing to agree to.
Hash a validated document, for the reason canonical/1 gives.
@spec sanitize(term(), Coelho.Schema.t()) :: map()
Turns any term into a document the schema accepts, without failing.
validate/2 is the boundary at the keyboard: it says no, and says where.
This is the boundary at the screen. Stored documents are not re-validated
when they are read — Coelho.Ecto.Type deliberately trusts the column —
so a row written under a looser schema, by a direct SQL write, or by a
version of the application that has since tightened its vocabulary, would
otherwise reach a public page unchecked.
Nothing is reported and nothing is raised: what falls outside the schema is
removed, and what is left is a document validate/2 accepts. A hostile
document becomes a poor document, never an unexpected rendering.
What removal means, from the gentlest repair to the harshest:
- a key the schema does not know is dropped
- an attribute failing its validator is dropped, so the schema default
applies — a heading claiming
level: 99renders as a level 1 heading - a mark that is unknown, not allowed here, or whose own attributes fail
is dropped, and the text it covered stays — a link with a
javascript:href becomes plain text - a node whose type is unknown, or whose content cannot satisfy its content expression, is dropped whole, along with the text inside it
- a document that cannot be repaired at all becomes
Coelho.empty/1
A document stamped with another schema version is repaired against this
schema and restamped with its version, rather than refused the way
validate/2 refuses it. Rendering a document written under an older
vocabulary badly beats rendering it as nothing; migrating it properly is
Coelho.migrate/2.
It is idempotent: a document that already validates comes back normalised and unchanged, and sanitising twice is sanitising once.
Coelho.Document.sanitize(row.body, MyApp.RichText.schema())
|> Coelho.Render.to_html(MyApp.RichText.schema())@spec text_length(term()) :: non_neg_integer()
The number of characters a writer typed.
This is the concatenation of the text nodes, nothing else: no bullet, no
blank line between paragraphs, no filename standing in for an attachment.
to_text/2 materialises all of those because full text search wants them,
and a length counted on its result rejects a document the editor still
shows as under the limit — with nothing on screen to explain the gap.
The browser half counts the same way, so the editor's counter and the server's check agree on the number.
@spec to_text(map(), Coelho.Schema.t()) :: String.t()
Extracts the plain text of a document, for full text search.
@spec validate(term(), Coelho.Schema.t()) :: {:ok, map()} | {:error, [Coelho.Document.Error.t()]}
Validates and normalises a document against a schema.
Returns the normalised document, or every error found. Paths in the
errors read from the root, as in content[0].attrs.href.