raxx v0.17.1 Raxx.HTTP1
Toolkit for parsing and serializing requests to HTTP/1.1 format.
The majority of functions return iolists and not compacted binaries.
To efficiently turn a list into a binart use :erlang.iolist_to_binary/1
Notes
content-length
The serializer does not add the content-length header for empty bodies. The rfc7230 says it SHOULD, but there are may cases where it must not be sent. This simplifies the serialization code.
It is probable that in the future Raxx.set_content_length/2
will be added.
And that it will be used by Raxx.set_body/2
This is because when parsing a message the content-length headers is kept.
Adding it to the Raxx.Request
struct will increase the cases when serialization and deserialization result in the exact same struct.
Property testing
Functionality in this module might be a good opportunity for property based testing. Elixir Outlaws convinced me to give it a try.
- Property of serialize then decode the head should end up with the same struct
- Propery of any number of splits in the binary should not change the output
Link to this section Summary
Functions
Extract the content from a buffer with transfer encoding chunked
Parse the head part of a request from a buffer
Parse the head of a response
Serialize io_data as a single chunk to be streamed
Serialize a request to wire format
Serialize a response to an iolist
Link to this section Types
body_read_state() :: {:complete, binary()} | {:bytes, non_neg_integer()} | :chunked
Link to this section Functions
Extract the content from a buffer with transfer encoding chunked
parse_request(binary(), [option]) :: {:ok, {Raxx.Request.t(), connection_status(), body_read_state(), binary()}} | {:error, term()} | {:more, :undefined} when option: {:scheme, atom()} | {:maximum_line_length, integer()}
Parse the head part of a request from a buffer.
The scheme is not part of a HTTP/1.1 request, yet it is part of a HTTP/2 request. When parsing a request the scheme the buffer was received by has to be given.
Options
scheme (required) Set the scheme of the
Raxx.Request
struct. This information is not contained in the data of a HTTP/1 request.maximum_headers_count Maximum number of headers allowed in the request.
maximum_line_length Maximum length (in bytes) of request line or any header line.
Examples
iex> "GET /path?qs HTTP/1.1\r\nhost: example.com\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
{%Raxx.Request{
authority: "example.com",
body: false,
headers: [{"accept", "text/plain"}],
method: :GET,
mount: [],
path: ["path"],
query: "qs",
raw_path: "/path",
scheme: :http
}, nil, {:complete, ""}, ""}}
iex> "GET /path?qs HTTP/1.1\r\nhost: example.com\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :https)
{:ok,
{%Raxx.Request{
authority: "example.com",
body: false,
headers: [{"accept", "text/plain"}],
method: :GET,
mount: [],
path: ["path"],
query: "qs",
raw_path: "/path",
scheme: :https
}, nil, {:complete, ""}, ""}}
iex> "POST /path HTTP/1.1\r\nhost: example.com\r\ntransfer-encoding: chunked\r\ncontent-type: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
{%Raxx.Request{
authority: "example.com",
body: true,
headers: [{"content-type", "text/plain"}],
method: :POST,
mount: [],
path: ["path"],
query: nil,
raw_path: "/path",
scheme: :http
}, nil, :chunked, ""}}
iex> "POST /path HTTP/1.1\r\nhost: example.com\r\ncontent-length: 13\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
{%Raxx.Request{
authority: "example.com",
body: true,
headers: [{"content-length", "13"}],
method: :POST,
mount: [],
path: ["path"],
query: nil,
raw_path: "/path",
scheme: :http
}, nil, {:bytes, 13}, ""}}
# Packet split in request line
iex> "GET /path?qs HT"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:more, "GET /path?qs HT"}
# Packet split in headers
iex> "GET / HTTP/1.1\r\nhost: exa"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:more, "GET / HTTP/1.1\r\nhost: exa"}
# Sending response
iex> "HTTP/1.1 204 No Content\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:invalid_line, "HTTP/1.1 204 No Content\r\n"}}
# Missing host header
iex> "GET / HTTP/1.1\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, :no_host_header}
# Duplicate host header
iex> "GET / HTTP/1.1\r\nhost: example.com\r\nhost: example2.com\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, :multiple_host_headers}
# Invalid content length header
iex> "GET / HTTP/1.1\r\nhost: example.com\r\ncontent-length: eleven\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, :invalid_content_length_header}
# Duplicate content length header
iex> "GET / HTTP/1.1\r\nhost: example.com\r\ncontent-length: 12\r\ncontent-length: 14\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, :multiple_content_length_headers}
# Invalid start line
iex> "!!!BAD_REQUEST_LINE\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:invalid_line, "!!!BAD_REQUEST_LINE\r\n"}}
# Invalid header line
iex> "GET / HTTP/1.1\r\n!!!BAD_HEADER\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:invalid_line, "!!!BAD_HEADER\r\n"}}
# Test connection status is extracted
iex> "GET / HTTP/1.1\r\nhost: example.com\r\nconnection: close\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
{%Raxx.Request{
authority: "example.com",
body: false,
headers: [{"accept", "text/plain"}],
method: :GET,
mount: [],
path: [],
query: nil,
raw_path: "/",
scheme: :http
}, :close, {:complete, ""}, ""}}
iex> "GET / HTTP/1.1\r\nhost: example.com\r\nconnection: keep-alive\r\naccept: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:ok,
{%Raxx.Request{
authority: "example.com",
body: false,
headers: [{"accept", "text/plain"}],
method: :GET,
mount: [],
path: [],
query: nil,
raw_path: "/",
scheme: :http
}, :keepalive, {:complete, ""}, ""}}
# Test line_length is limited
# "GET /" + "HTTP/1.1\r\n" = 15
iex> path = "/" <> String.duplicate("a", 985)
...> "GET #{path} HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:line_length_limit_exceeded, :request_line}}
iex> path = "/" <> String.duplicate("a", 984)
...> "GET #{path} HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
...> |> elem(0)
:more
iex> path = "/" <> String.duplicate("a", 1984)
...> "GET #{path} HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http, maximum_line_length: 2000)
...> |> elem(0)
:more
iex> "GET / HTTP/1.1\r\nhost: #{String.duplicate("a", 993)}\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
{:error, {:line_length_limit_exceeded, :header_line}}
iex> "GET / HTTP/1.1\r\nhost: #{String.duplicate("a", 992)}\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http)
...> |> elem(0)
:more
iex> "GET / HTTP/1.1\r\nhost: #{String.duplicate("a", 1992)}\r\n"
...> |> Raxx.HTTP1.parse_request(scheme: :http, maximum_line_length: 2000)
...> |> elem(0)
:more
parse_response(binary(), [option]) :: {:ok, {Raxx.Response.t(), connection_status(), body_read_state(), binary()}} | {:error, term()} | {:more, :undefined} when option: {:maximum_line_length, integer()}
Parse the head of a response.
A scheme option is not given to this parser because the scheme not a requirement in HTTP/1 or HTTP/2
Options
maximum_headers_count Maximum number of headers allowed in the request.
maximum_line_length Maximum length (in bytes) of request line or any header line.
Examples
iex> "HTTP/1.1 204 No Content\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
status: 204,
headers: [{"foo", "bar"}],
body: false
}, nil, {:complete, ""}, ""}}
iex> "HTTP/1.1 200 OK\r\ncontent-length: 13\r\ncontent-type: text/plain\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
status: 200,
headers: [{"content-length", "13"}, {"content-type", "text/plain"}],
body: true
}, nil, {:bytes, 13}, ""}}
iex> "HTTP/1.1 204 No Con"
...> |> Raxx.HTTP1.parse_response()
{:more, :undefined}
iex> "HTTP/1.1 204 No Content\r\nfo"
...> |> Raxx.HTTP1.parse_response()
{:more, :undefined}
# Request given
iex> "GET / HTTP/1.1\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:invalid_line, "GET / HTTP/1.1\r\n"}}
iex> "!!!BAD_STATUS_LINE\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:invalid_line, "!!!BAD_STATUS_LINE\r\n"}}
iex> "HTTP/1.1 204 No Content\r\n!!!BAD_HEADER\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:invalid_line, "!!!BAD_HEADER\r\n"}}
iex> "HTTP/1.1 204 No Content\r\nconnection: close\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
status: 204,
headers: [{"foo", "bar"}],
body: false
}, :close, {:complete, ""}, ""}}
iex> "HTTP/1.1 204 No Content\r\nconnection: keep-alive\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
status: 204,
headers: [{"foo", "bar"}],
body: false
}, :keepalive, {:complete, ""}, ""}}
# Test exceptional case when server returns Title case values
iex> "HTTP/1.1 204 No Content\r\nconnection: Close\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
status: 204,
headers: [{"foo", "bar"}],
body: false
}, :close, {:complete, ""}, ""}}
# Test exceptional case when server uses Title case values
iex> "HTTP/1.1 204 No Content\r\nconnection: Keep-alive\r\nfoo: bar\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:ok, {%Raxx.Response{
status: 204,
headers: [{"foo", "bar"}],
body: false
}, :keepalive, {:complete, ""}, ""}}
# Invalid connection header
iex> "HTTP/1.1 204 No Content\r\nconnection: Invalid\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, :invalid_connection_header}
# duplicate connection header
iex> "HTTP/1.1 204 No Content\r\nconnection: close\r\nconnection: keep-alive\r\n\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, :multiple_connection_headers}
# Test line_length is limited
# "HTTP/1.1 204 " + newlines = 15
iex> reason_phrase = String.duplicate("A", 986)
...> "HTTP/1.1 204 #{reason_phrase}\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:line_length_limit_exceeded, :status_line}}
iex> reason_phrase = String.duplicate("A", 985)
...> "HTTP/1.1 204 #{reason_phrase}\r\n"
...> |> Raxx.HTTP1.parse_response()
...> |> elem(0)
:more
iex> reason_phrase = String.duplicate("A", 1985)
...> "HTTP/1.1 204 #{reason_phrase}\r\n"
...> |> Raxx.HTTP1.parse_response(maximum_line_length: 2000)
...> |> elem(0)
:more
iex> "HTTP/1.1 204 No Content\r\nfoo: #{String.duplicate("a", 994)}\r\n"
...> |> Raxx.HTTP1.parse_response()
{:error, {:line_length_limit_exceeded, :header_line}}
iex> "HTTP/1.1 204 No Content\r\nfoo: #{String.duplicate("a", 993)}\r\n"
...> |> Raxx.HTTP1.parse_response()
...> |> elem(0)
:more
iex> "HTTP/1.1 204 No Content\r\nfoo: #{String.duplicate("a", 1993)}\r\n"
...> |> Raxx.HTTP1.parse_response(maximum_line_length: 2000)
...> |> elem(0)
:more
# Test maximum number of headers is limited
iex> "HTTP/1.1 204 No Content\r\n#{String.duplicate("foo: bar\r\n", 101)}"
...> |> Raxx.HTTP1.parse_response()
{:error, {:header_count_exceeded, 100}}
# Test maximum number of headers is limited
iex> "HTTP/1.1 204 No Content\r\n#{String.duplicate("foo: bar\r\n", 2)}"
...> |> Raxx.HTTP1.parse_response(maximum_headers_count: 1)
{:error, {:header_count_exceeded, 1}}
Serialize io_data as a single chunk to be streamed.
Example
iex> Raxx.HTTP1.serialize_chunk("hello")
...> |> to_string()
"5\r\nhello\r\n"
iex> Raxx.HTTP1.serialize_chunk("")
...> |> to_string()
"0\r\n\r\n"
serialize_request(Raxx.Request.t(), [{:connection, connection_status()}]) :: {iodata(), body_read_state()}
Serialize a request to wire format
NOTE set_body should add content-length otherwise we don’t know if to delete it to match on other end, when serializing
https://tools.ietf.org/html/rfc7230#section-5.4
Since the Host field-value is critical information for handling a request, a user agent SHOULD generate Host as the first header field following the request-line.
Examples
iex> request = Raxx.request(:GET, "http://example.com/path?qs")
...> |> Raxx.set_header("accept", "text/plain")
...> {head, body} = Raxx.HTTP1.serialize_request(request)
...> :erlang.iolist_to_binary(head)
"GET /path?qs HTTP/1.1\r\nhost: example.com\r\naccept: text/plain\r\n\r\n"
iex> body
{:complete, ""}
iex> request = Raxx.request(:POST, "https://example.com")
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body(true)
...> {head, body} = Raxx.HTTP1.serialize_request(request)
...> :erlang.iolist_to_binary(head)
"POST / HTTP/1.1\r\nhost: example.com\r\ntransfer-encoding: chunked\r\ncontent-type: text/plain\r\n\r\n"
iex> body
:chunked
iex> request = Raxx.request(:POST, "https://example.com")
...> |> Raxx.set_header("content-length", "13")
...> |> Raxx.set_body(true)
...> {head, body} = Raxx.HTTP1.serialize_request(request)
...> :erlang.iolist_to_binary(head)
"POST / HTTP/1.1\r\nhost: example.com\r\ncontent-length: 13\r\n\r\n"
iex> body
{:bytes, 13}
https://tools.ietf.org/html/rfc7230#section-6.1
A client that does not support persistent connections MUST send the “close” connection option in every request message.
iex> request = Raxx.request(:GET, "http://example.com/")
...> |> Raxx.set_header("accept", "text/plain")
...> {head, _body} = Raxx.HTTP1.serialize_request(request, connection: :close)
...> :erlang.iolist_to_binary(head)
"GET / HTTP/1.1\r\nhost: example.com\r\nconnection: close\r\naccept: text/plain\r\n\r\n"
iex> request = Raxx.request(:GET, "http://example.com/")
...> |> Raxx.set_header("accept", "text/plain")
...> {head, _body} = Raxx.HTTP1.serialize_request(request, connection: :keepalive)
...> :erlang.iolist_to_binary(head)
"GET / HTTP/1.1\r\nhost: example.com\r\nconnection: keep-alive\r\naccept: text/plain\r\n\r\n"
serialize_response(Raxx.Response.t(), [{:connection, connection_status()}]) :: {iolist(), body_read_state()}
Serialize a response to an iolist
Because of HEAD requests we should keep body separate
Examples
iex> response = Raxx.response(200)
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body("Hello, World!")
...> {head, body} = Raxx.HTTP1.serialize_response(response)
...> :erlang.iolist_to_binary(head)
"HTTP/1.1 200 OK\r\ncontent-type: text/plain\r\ncontent-length: 13\r\n\r\n"
iex> body
{:complete, "Hello, World!"}
iex> response = Raxx.response(200)
...> |> Raxx.set_header("content-length", "13")
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body(true)
...> {head, body} = Raxx.HTTP1.serialize_response(response)
...> :erlang.iolist_to_binary(head)
"HTTP/1.1 200 OK\r\ncontent-length: 13\r\ncontent-type: text/plain\r\n\r\n"
iex> body
{:bytes, 13}
iex> response = Raxx.response(200)
...> |> Raxx.set_header("content-type", "text/plain")
...> |> Raxx.set_body(true)
...> {head, body} = Raxx.HTTP1.serialize_response(response)
...> :erlang.iolist_to_binary(head)
"HTTP/1.1 200 OK\r\ntransfer-encoding: chunked\r\ncontent-type: text/plain\r\n\r\n"
iex> body
:chunked
> A server MUST NOT send a Content-Length header field in any response
> with a status code of 1xx (Informational) or 204 (No Content). A
> server MUST NOT send a Content-Length header field in any 2xx
> (Successful) response to a CONNECT request (Section 4.3.6 of
> [RFC7231]).
iex> Raxx.response(204)
...> |> Raxx.set_header("foo", "bar")
...> |> Raxx.HTTP1.serialize_response()
...> |> elem(0)
...> |> :erlang.iolist_to_binary()
"HTTP/1.1 204 No Content\r\nfoo: bar\r\n\r\n"
https://tools.ietf.org/html/rfc7230#section-6.1
A server that does not support persistent connections MUST send the “close” connection option in every response message that does not have a 1xx (Informational) status code.
iex> Raxx.response(204)
...> |> Raxx.set_header("foo", "bar")
...> |> Raxx.HTTP1.serialize_response(connection: :close)
...> |> elem(0)
...> |> :erlang.iolist_to_binary()
"HTTP/1.1 204 No Content\r\nconnection: close\r\nfoo: bar\r\n\r\n"
iex> Raxx.response(204)
...> |> Raxx.set_header("foo", "bar")
...> |> Raxx.HTTP1.serialize_response(connection: :keepalive)
...> |> elem(0)
...> |> :erlang.iolist_to_binary()
"HTTP/1.1 204 No Content\r\nconnection: keep-alive\r\nfoo: bar\r\n\r\n"