raxx v0.15.10 Raxx.HTTP1

Toolkit for parsing and serializing requests to HTTP/1.1 format.

The majority of functions return iolists and not compacted binaries. To efficiently turn a list into a binart use :erlang.iolist_to_binary/1

Notes

content-length

The serializer does not add the content-length header for empty bodies. The rfc7230 says it SHOULD, but there are may cases where it must not be sent. This simplifies the serialization code.

It is probable that in the future Raxx.set_content_length/2 will be added. And that it will be used by Raxx.set_body/2

This is because when parsing a message the content-length headers is kept. Adding it to the Raxx.Request struct will increase the cases when serialization and deserialization result in the exact same struct.

Property testing

Functionality in this module might be a good opportunity for property based testing. Elixir Outlaws convinced me to give it a try.

  • Property of serialize then decode the head should end up with the same struct
  • Propery of any number of splits in the binary should not change the output

Link to this section Summary

Functions

Extract the content from a buffer with transfer encoding chunked

Parse the head part of a request from a buffer

Parse the head of a response

Serialize io_data as a single chunk to be streamed

Serialize a request to wire format

Serialize a response to an iolist

Link to this section Types

Link to this type body_read_state()
body_read_state() ::
  {:complete, binary()} | {:bytes, non_neg_integer()} | :chunked
Link to this type connection_status()
connection_status() :: nil | :close | :keepalive

Link to this section Functions

Link to this function parse_chunk(buffer)
parse_chunk(binary()) :: {:ok, {binary() | nil, binary()}}

Extract the content from a buffer with transfer encoding chunked

Link to this function parse_request(buffer, options)
parse_request(binary(), [option]) ::
  {:ok, {Raxx.Request.t(), connection_status(), body_read_state(), binary()}}
  | {:error, term()}
  | {:more, :undefined}
when option: {:scheme, atom()} | {:maximum_line_length, integer()}

Parse the head part of a request from a buffer.

The scheme is not part of a HTTP/1.1 request, yet it is part of a HTTP/2 request. When parsing a request the scheme the buffer was received by has to be given.

Options

  • scheme (required) Set the scheme of the Raxx.Request struct. This information is not contained in the data of a HTTP/1 request.

  • maximum_headers_count Maximum number of headers allowed in the request.

  • maximum_line_length Maximum length (in bytes) of request line or any header line.

Examples

iex> "GET /path?qs HTTP/1.1\r\nhost: example.com\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
 {%Raxx.Request{
   authority: "example.com",
   body: false,
   headers: [{"accept", "text/plain"}],
   method: :GET,
   mount: [],
   path: ["path"],
   query: "qs",
   raw_path: "/path",
   scheme: :http
 }, nil, {:complete, ""}, ""}}

iex> "GET /path?qs HTTP/1.1\r\nhost: example.com\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :https)
{:ok,
 {%Raxx.Request{
   authority: "example.com",
   body: false,
   headers: [{"accept", "text/plain"}],
   method: :GET,
   mount: [],
   path: ["path"],
   query: "qs",
   raw_path: "/path",
   scheme: :https
 }, nil, {:complete, ""}, ""}}

iex> "POST /path HTTP/1.1\r\nhost: example.com\r\ntransfer-encoding: chunked\r\ncontent-type: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
 {%Raxx.Request{
   authority: "example.com",
   body: true,
   headers: [{"content-type", "text/plain"}],
   method: :POST,
   mount: [],
   path: ["path"],
   query: nil,
   raw_path: "/path",
   scheme: :http
 }, nil, :chunked, ""}}

iex> "POST /path HTTP/1.1\r\nhost: example.com\r\ncontent-length: 13\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
 {%Raxx.Request{
   authority: "example.com",
   body: true,
   headers: [{"content-length", "13"}],
   method: :POST,
   mount: [],
   path: ["path"],
   query: nil,
   raw_path: "/path",
   scheme: :http
 }, nil, {:bytes, 13}, ""}}

iex> "GET /path?qs HT"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:more, "GET /path?qs HT"}

iex> "GET /path?qs HTTP/1.1\r\nhost: exa"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:more, "GET /path?qs HTTP/1.1\r\nhost: exa"}

# Missing host header
iex> "GET /path?qs HTTP/1.1\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, :no_host_header}

# Invalid start line
iex> "!!!BAD_REQUEST_LINE\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:invalid_line, "!!!BAD_REQUEST_LINE\r\n"}}

# Invalid header line
iex> "GET / HTTP/1.1\r\n!!!BAD_HEADER\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:invalid_line, "!!!BAD_HEADER\r\n"}}

# Test connection status is extracted
iex> "GET /path?qs HTTP/1.1\r\nhost: example.com\r\nconnection: close\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
 {%Raxx.Request{
   authority: "example.com",
   body: false,
   headers: [{"accept", "text/plain"}],
   method: :GET,
   mount: [],
   path: ["path"],
   query: "qs",
   raw_path: "/path",
   scheme: :http
 }, :close, {:complete, ""}, ""}}

 iex> "GET /path?qs HTTP/1.1\r\nhost: example.com\r\nconnection: keep-alive\r\naccept: text/plain\r\n\r\n"
 ...> |> Raxx.HTTP1.parse_request(scheme: :http)
 {:ok,
  {%Raxx.Request{
    authority: "example.com",
    body: false,
    headers: [{"accept", "text/plain"}],
    method: :GET,
    mount: [],
    path: ["path"],
    query: "qs",
    raw_path: "/path",
    scheme: :http
  }, :keepalive, {:complete, ""}, ""}}

# Test line_length is limited
# "GET /" +  "HTTP/1.1\r\n" = 15
iex> path = "/" <> String.duplicate("a", 985)
...> "GET #{path} HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:line_length_limit_exceeded, :request_line}}

iex> path = "/" <> String.duplicate("a", 984)
...> {:more, _} = "GET #{path} HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
...> :ok
:ok

iex> path = "/" <> String.duplicate("a", 1984)
...> {:more, _} = "GET #{path} HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http, maximum_line_length: 2000)
...> :ok
:ok

iex> "GET / HTTP/1.1\r\nhost: #{String.duplicate("a", 993)}\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:line_length_limit_exceeded, :header_line}}

iex> {:more, _} = "GET / HTTP/1.1\r\nhost: #{String.duplicate("a", 992)}\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
...> :ok
:ok

iex> {:more, _} = "GET / HTTP/1.1\r\nhost: #{String.duplicate("a", 1992)}\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http, maximum_line_length: 2000)
...> :ok
:ok
Link to this function parse_response(buffer, options \\ [])
parse_response(binary(), [option]) ::
  {:ok, {Raxx.Response.t(), connection_status(), body_read_state(), binary()}}
  | {:error, term()}
  | {:more, :undefined}
when option: {:maximum_line_length, integer()}

Parse the head of a response.

A scheme option is not given to this parser because the scheme not a requirement in HTTP/1 or HTTP/2

Options

  • maximum_headers_count Maximum number of headers allowed in the request.

  • maximum_line_length Maximum length (in bytes) of request line or any header line.

Examples

iex> "HTTP/1.1 204 No Content\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
  status: 204,
  headers: [{"foo", "bar"}],
  body: false
}, nil, {:complete, ""}, ""}}

iex> "HTTP/1.1 200 OK\r\ncontent-length: 13\r\ncontent-type: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
  status: 200,
  headers: [{"content-length", "13"}, {"content-type", "text/plain"}],
  body: true
}, nil, {:bytes, 13}, ""}}

iex> "HTTP/1.1 204 No Con"
...> |> Raxx.HTTP1.parse_response()
{:more, :undefined}

iex> "HTTP/1.1 204 No Content\r\nfo"
...> |> Raxx.HTTP1.parse_response()
{:more, :undefined}

iex> "!!!BAD_STATUS_LINE\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:invalid_line, "!!!BAD_STATUS_LINE\r\n"}}

iex> "HTTP/1.1 204 No Content\r\n!!!BAD_HEADER\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:invalid_line, "!!!BAD_HEADER\r\n"}}

iex> "HTTP/1.1 204 No Content\r\nconnection: close\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
  status: 204,
  headers: [{"foo", "bar"}],
  body: false
}, :close, {:complete, ""}, ""}}

iex> "HTTP/1.1 204 No Content\r\nconnection: keep-alive\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
  status: 204,
  headers: [{"foo", "bar"}],
  body: false
}, :keepalive, {:complete, ""}, ""}}

# Test exceptional case when server returns Title case values
iex> "HTTP/1.1 204 No Content\r\nconnection: Close\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
  status: 204,
  headers: [{"foo", "bar"}],
  body: false
}, :close, {:complete, ""}, ""}}

# Test exceptional case when server uses Title case values
iex> "HTTP/1.1 204 No Content\r\nconnection: Keep-alive\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
  status: 204,
  headers: [{"foo", "bar"}],
  body: false
}, :keepalive, {:complete, ""}, ""}}

# Test line_length is limited
# "HTTP/1.1 204 " + newlines = 15
iex> reason_phrase = String.duplicate("A", 986)
...> "HTTP/1.1 204 #{reason_phrase}\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:line_length_limit_exceeded, :status_line}}

iex> reason_phrase = String.duplicate("A", 985)
...> {:more, _} = "HTTP/1.1 204 #{reason_phrase}\r\n"
...> |> Raxx.HTTP1.parse_response()
...> :ok
:ok

iex> reason_phrase = String.duplicate("A", 1985)
...> {:more, _} = "HTTP/1.1 204 #{reason_phrase}\r\n"
...> |> Raxx.HTTP1.parse_response(maximum_line_length: 2000)
...> :ok
:ok

iex> "HTTP/1.1 204 No Content\r\nfoo: #{String.duplicate("a", 994)}\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:line_length_limit_exceeded, :header_line}}

iex> {:more, _} = "HTTP/1.1 204 No Content\r\nfoo: #{String.duplicate("a", 993)}\r\n"
...> |> Raxx.HTTP1.parse_response()
...> :ok
:ok

iex> {:more, _} = "HTTP/1.1 204 No Content\r\nfoo: #{String.duplicate("a", 1993)}\r\n"
...> |> Raxx.HTTP1.parse_response(maximum_line_length: 2000)
...> :ok
:ok

# Test maximum number of headers is limited
iex> "HTTP/1.1 204 No Content\r\n#{String.duplicate("foo: bar\r\n", 101)}"
...> |> Raxx.HTTP1.parse_response()
{:error, :header_count_exceeded}

# Test maximum number of headers is limited
iex> "HTTP/1.1 204 No Content\r\n#{String.duplicate("foo: bar\r\n", 2)}"
...> |> Raxx.HTTP1.parse_response(maximum_headers_count: 1)
{:error, :header_count_exceeded}
Link to this function serialize_chunk(data)
serialize_chunk(iodata()) :: iodata()

Serialize io_data as a single chunk to be streamed.

Example

iex> Raxx.HTTP1.serialize_chunk("hello")
...> |> to_string()
"5\r\nhello\r\n"

iex> Raxx.HTTP1.serialize_chunk("")
...> |> to_string()
"0\r\n\r\n"
Link to this function serialize_request(request, options \\ [])
serialize_request(Raxx.Request.t(), [{:connection, connection_status()}]) ::
  {iodata(), body_read_state()}

Serialize a request to wire format

NOTE set_body should add content-length otherwise we don’t know if to delete it to match on other end, when serializing

https://tools.ietf.org/html/rfc7230#section-5.4

Since the Host field-value is critical information for handling a request, a user agent SHOULD generate Host as the first header field following the request-line.

Examples

iex> request = Raxx.request(:GET, "http://example.com/path?qs")
...> |> Raxx.set_header("accept", "text/plain")
...> {head, body} =  Raxx.HTTP1.serialize_request(request)
...> :erlang.iolist_to_binary(head)
"GET /path?qs HTTP/1.1\r\nhost: example.com\r\naccept: text/plain\r\n\r\n"
iex> body
{:complete, ""}

iex> request = Raxx.request(:POST, "https://example.com")
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body(true)
...> {head, body} =  Raxx.HTTP1.serialize_request(request)
...> :erlang.iolist_to_binary(head)
"POST / HTTP/1.1\r\nhost: example.com\r\ntransfer-encoding: chunked\r\ncontent-type: text/plain\r\n\r\n"
iex> body
:chunked

iex> request = Raxx.request(:POST, "https://example.com")
...> |> Raxx.set_header("content-length", "13")
...> |> Raxx.set_body(true)
...> {head, body} =  Raxx.HTTP1.serialize_request(request)
...> :erlang.iolist_to_binary(head)
"POST / HTTP/1.1\r\nhost: example.com\r\ncontent-length: 13\r\n\r\n"
iex> body
{:bytes, 13}

https://tools.ietf.org/html/rfc7230#section-6.1

A client that does not support persistent connections MUST send the “close” connection option in every request message.

iex> request = Raxx.request(:GET, "http://example.com/")
...> |> Raxx.set_header("accept", "text/plain")
...> {head, _body} =  Raxx.HTTP1.serialize_request(request, connection: :close)
...> :erlang.iolist_to_binary(head)
"GET / HTTP/1.1\r\nhost: example.com\r\nconnection: close\r\naccept: text/plain\r\n\r\n"

iex> request = Raxx.request(:GET, "http://example.com/")
...> |> Raxx.set_header("accept", "text/plain")
...> {head, _body} =  Raxx.HTTP1.serialize_request(request, connection: :keepalive)
...> :erlang.iolist_to_binary(head)
"GET / HTTP/1.1\r\nhost: example.com\r\nconnection: keep-alive\r\naccept: text/plain\r\n\r\n"
Link to this function serialize_response(response, options \\ [])
serialize_response(Raxx.Response.t(), [{:connection, connection_status()}]) ::
  {iolist(), body_read_state()}

Serialize a response to an iolist

Because of HEAD requests we should keep body separate

Examples

iex> response = Raxx.response(200)
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body("Hello, World!")
...> {head, body} =  Raxx.HTTP1.serialize_response(response)
...> :erlang.iolist_to_binary(head)
"HTTP/1.1 200 OK\r\ncontent-length: 13\r\ncontent-type: text/plain\r\n\r\n"
iex> body
{:complete, "Hello, World!"}

iex> response = Raxx.response(200)
...> |> Raxx.set_header("content-length", "13")
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body(true)
...> {head, body} =  Raxx.HTTP1.serialize_response(response)
...> :erlang.iolist_to_binary(head)
"HTTP/1.1 200 OK\r\ncontent-length: 13\r\ncontent-type: text/plain\r\n\r\n"
iex> body
{:bytes, 13}

iex> response = Raxx.response(200)
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body(true)
...> {head, body} =  Raxx.HTTP1.serialize_response(response)
...> :erlang.iolist_to_binary(head)
"HTTP/1.1 200 OK\r\ntransfer-encoding: chunked\r\ncontent-type: text/plain\r\n\r\n"
iex> body
:chunked

> A server MUST NOT send a Content-Length header field in any response
> with a status code of 1xx (Informational) or 204 (No Content).  A
> server MUST NOT send a Content-Length header field in any 2xx
> (Successful) response to a CONNECT request (Section 4.3.6 of
> [RFC7231]).

iex> Raxx.response(204)
...> |> Raxx.set_header("foo", "bar")
...> |> Raxx.HTTP1.serialize_response()
...> |> elem(0)
...> |> :erlang.iolist_to_binary()
"HTTP/1.1 204 No Content\r\nfoo: bar\r\n\r\n"

https://tools.ietf.org/html/rfc7230#section-6.1

A server that does not support persistent connections MUST send the “close” connection option in every response message that does not have a 1xx (Informational) status code.

iex> Raxx.response(204)
...> |> Raxx.set_header("foo", "bar")
...> |> Raxx.HTTP1.serialize_response(connection: :close)
...> |> elem(0)
...> |> :erlang.iolist_to_binary()
"HTTP/1.1 204 No Content\r\nconnection: close\r\nfoo: bar\r\n\r\n"

iex> Raxx.response(204)
...> |> Raxx.set_header("foo", "bar")
...> |> Raxx.HTTP1.serialize_response(connection: :keepalive)
...> |> elem(0)
...> |> :erlang.iolist_to_binary()
"HTTP/1.1 204 No Content\r\nconnection: keep-alive\r\nfoo: bar\r\n\r\n"