The Buffer provides blocking writes and blocking reads to a pre-defined I/O buffer, which is suitable for applying backpressure to content loaded from servers.
The Buffer is used by the URL source. It is implemented as a state machine with the following states:
- Buffering: Where the buffer is in use. The Buffer is always in this state, but tracks the number of bytes already buffered. When it is not full, further calls to load data will return immediately; if the buffer is full, calls will not be handled until the buffer has been drained via the read call.
- Finished: Where the buffer has been emptied and a call has marked the buffer as Finished, which is expected behaviour from the URL Source Reader. This is necessary so we can distinguish an interrupted connection from a correctly finished one.
The use of the Buffer in the URL Source Reader is as follows:
The URL Source Reader creates the Buffer, and continuously writes to it by making the
{:data, chunk}calls, until it is blocked due to the buffer filling up, in which case further calls are postponed (blocked on caller side)The URL Source would read from the buffer whenever the Encoder calls
read/1, which in time empties the buffer, allowing further{:data, chunk}calls to be unblockedThe URL Source Reader would eventually, when the underlying Req request has finished properly, send a
:finishcall to the Buffer, which will cause the Buffer to return:eofwhen the URL Source eventually attempts to read from the Buffer. This is all properly sequenced, because all previous{:data, chunk}events will have to be processed first, before the:finishevent can be processed.