ExAws.S3.download_file

You're seeing just the function download_file, go back to ExAws.S3 module for more information.
Link to this function

download_file(bucket, path, dest, opts \\ [])

View Source

Specs

download_file(
  bucket :: binary(),
  path :: binary(),
  dest :: :memory | binary(),
  opts :: download_file_opts()
) :: ExAws.S3.Download.t()

Download an S3 object to a file.

This operation downloads multiple parts of an S3 object concurrently, allowing you to maximize throughput.

Defaults to a concurrency of 8, chunk size of 1MB, and a timeout of 1 minute.

Streaming to memory

In order to use ExAws.stream!/2, the third dest parameter must be set to :memory. An example would be like the following:

ExAws.S3.download_file("example-bucket", "path/to/file.txt", :memory)
|> ExAws.stream!()

Note that this won't start fetching anything immediately since it returns an Elixir Stream.

Streaming by line

Streaming by line can be done with Stream.chunk_while/4. Here is an example:

# Returns a stream which grabs chunks of data from S3 as specified in `opts`
# but processes the stream line by line. For example, the default chunk
# size of 1MB means requests for bytes from S3 will ask for 1MB sizes (to be downloaded)
# however each element of the stream will be a single line.
def generate_stream(bucket, file, opts \\ []) do
  bucket
  |> ExAws.S3.download_file(file, :memory, opts)
  |> ExAws.stream!()
  # Uncomment if you need to gunzip (and add dependency :stream_gzip)
  # |> StreamGzip.gunzip()
  |> Stream.chunk_while("", &chunk_fun/2, &after_fun/1)
end

def chunk_fun(chunk, acc) do
  split_chunk(acc, chunk) || split_chunk(chunk, "")
end

defp split_chunk("", _append_remaining), do: nil
defp split_chunk(string, append_remaining) do
  case String.split(string, "\n", parts: 2) do
    [l] ->
      {:cont, l}
    [l, remaining] ->
      {:cont, l, remaining <> append_remaining}
  end
end

def after_fun(""), do: {:cont, ""}
def after_fun(acc), do: {:cont, acc, ""}