The PostgreSQL frontend/backend protocol, version 3.0, as a pure function of bytes.

Terms in, wire bytes out for the messages a client sends. Wire bytes in, terms out for the messages a server sends. This module does no I/O, starts no process and keeps no state, so every message can be tested exhaustively without a database and without a socket.

Framing

frame/1 takes whatever bytes have accumulated and returns every complete message in them together with the bytes left over:

{Messages, Rest} = minato_protocol:frame(<<Carried/binary, Received/binary>>).

That signature is the point. A backend message is a type byte, a 32 bit length and a payload, so it is tempting to read the five byte header, then read the payload, then repeat. That is two reads per message and it scales with the number of rows. Reading whatever the socket has and carrying the remainder forward reads a whole result set in a handful of calls instead.

minato_framer_bench measures both against each other over a loopback socket. A 5000 row result set of eight columns takes 61 reads bulk against 10,006 reads header-then-payload, and 5.3 to 9.1 times less wall clock over repeated runs on OTP 28 and 29. The read counts are exact and the multiple is not: it moves with the machine, and on a real network it moves with the round trip too.

Carry-over only works if a decoder can say "not yet" without failing, so decode/1 answers incomplete for any buffer that stops short of a whole message, at any byte boundary, and frame/1 hands those bytes back untouched.

Total over arbitrary bytes

frame/1 and decode/1 never fail in any way other than error({minato_protocol, t:report/0}). Every other input, including bytes that are not a PostgreSQL stream at all, yields messages and a remainder. That matters because the bytes reaching the framer are whatever the far end sent, and a client that can be made to raise badarg from a length field has handed the far end a way to choose which exception the caller sees.

Bounds

A declared length longer than max_message_length raises rather than asking for more bytes. Without that a five byte header claiming four gigabytes would put the framer into incomplete, and a reader loop doing as it is told would accumulate until the node died. Refusing the header ends the connection instead, which is the correct outcome for a server that cannot be believed.

The default is 64 MiB. It is above any row an application selects on purpose and far below what would hurt a node, and the failure it guards against is not a large result set: a result set arrives as many messages, and only one message has to fit under the limit. Raise it for a single very large bytea or text column, since PostgreSQL caps one value, and therefore one message, at just under 1 GiB. A declared length above that ceiling cannot have come from PostgreSQL at all.

Binary retention

The messages share the buffer's bytes. {data_row, Payload} hands back the row's bytes rather than a copy, which is what makes framing a large result set cheap: nothing in a five thousand row buffer is copied on the way out, and the payload is consumed by decode_row/2 and dropped a row later.

The cost of that is the usual one. A sub-binary keeps the whole buffer it points into alive, so a column value the application keeps holds the megabyte it arrived in. binary:copy/1 on a value being kept for longer than the query is the caller's fix, and it is the caller's because only the caller knows which values those are. Copying them all here would put a copy of the entire result set on the hot path to protect the few values that outlive it.

The remainder is the one place where the framer copies, because it is the one sub-binary whose lifetime the framer knows. It is held until the next read completes, which on an idle pooled connection is unbounded, and it is by construction small while the buffer it points into is not: a three byte partial header retaining a 64 KiB read, across a hundred pooled connections, is megabytes of buffers kept alive by bytes nobody is waiting on. So a remainder is copied when it is at most a quarter of its buffer, which bounds the copy at a third of what it releases. A larger remainder is left alone, because copying it would cost more than it frees, and a half arrived multi-megabyte row would then be copied again on every read as it grew.

Errors from the server are data

{error_response, t:fields/0} is an ordinary decoded message, not an exception. A failing statement is a normal outcome of running SQL and the caller has to branch on it. server_error/1 wraps the fields in the tagged form consumers match on:

{error, {pgsql_error, #{code := ~"23505", constraint := ~"users_email_key"}}}

Keys are atoms and values are binaries, code is the SQLSTATE, and code and message are always present.

A malformed message is the opposite case and raises error({minato_protocol, t:report/0}). Bytes that do not fit the format the server itself announced are a bug, not a condition to branch on.

Rows

{data_row, Payload} carries the row's bytes undivided, because a DataRow cannot be read on its own: the type OID and wire format of each column live in the RowDescription that preceded it. decoder/2 turns that description into the answer once, and decode_row/2 walks the payload cutting and decoding each column in the same pass - which is the whole reason the payload is not split here.

{[{row_description, Fields}], _} = minato_protocol:frame(Bytes),
{[{data_row, Payload}], _} = minato_protocol:frame(MoreBytes),
Decoder = minato_protocol:decoder(Fields, #{}),
minato_protocol:decode_row(Decoder, Payload).

columns/1 splits a payload into the raw column bytes for a caller that wants them without decoding.

Options

#{return_rows_as_maps => boolean(),
  column_name_as_atom => boolean(),
  uuid_format => string | binary,
  datetime_format => datetime | microseconds,
  numeric_format => binary | float}

return_rows_as_maps defaults to false, giving a tuple per row. true gives a map keyed by column name. column_name_as_atom defaults to false and only affects those map keys.

column_name_as_atom is off by default because a column name is not always something the application wrote. A query built at run time can alias a column after a value, and every distinct alias would then make an atom that is never collected. Turn it on for queries whose column names you chose.

uuid_format, datetime_format and numeric_format are passed through to minato_codec unchanged.

Messages not modelled

Copy-mode messages, function calls and protocol version negotiation decode to {unsupported, TypeByte, Payload} rather than raising. Every backend message is length prefixed, so a message minato does not model can still be stepped over without losing the stream.