View Source Stdio.Stream (stdio v0.4.4)

Stream standard I/O from system processes

Summary

Functions

Rate limits a stream by discarding elements exceeding a threshold for the remainder of the window.

Rate limits a stream by calling a function on each event.

Apply back pressure on Stdio.stream!/1 or Stdio.pipe!/2 using ratelimit/4 by blocking further reads if the rate of stdout/stderr exceeds the threshold.

Rate limit output of Stdio.stream!/1 or Stdio.pipe!/2 using ratelimit/4 by dropping events if the rate of stdout/stderr exceeds the threshold.

Combine standard error into the standard output of Stdio.stream!/1 or Stdio.pipe!/2 and emit a list of binaries.

Types

@type t() :: %Stdio.Stream{
  flush_timeout: 0 | :infinity,
  onexit: (Stdio.ProcessTree.t() -> boolean()),
  process: Stdio.ProcessTree.t() | nil,
  status: :running | :flush | :flushing,
  stream_pid: pid() | nil
}

Functions

Link to this function

ratelimit(stream, limit, ms)

View Source
@spec ratelimit(Enumerable.t(), pos_integer(), pos_integer()) :: Enumerable.t()

Rate limits a stream by discarding elements exceeding a threshold for the remainder of the window.

ratelimit/3 works with any Enumerable.t. For working with streams generated by Stdio, see ratelimit/4.

Examples

iex> Stream.unfold(1_000, fn 0 -> nil; n -> {n, n-1} end)
...> |> Stdio.Stream.ratelimit(2, 10_000)
...> |> Enum.to_list()
[1000, 999]
Link to this function

ratelimit(stream, limit, ms, limitfun)

View Source
@spec ratelimit(
  Enumerable.t(),
  pos_integer(),
  pos_integer(),
  (term(), acc -> {Enumerable.t(), acc} | {:halt, term()})
) :: Enumerable.t()
when acc: %{t: integer(), limit: non_neg_integer(), ms: non_neg_integer()}

Rate limits a stream by calling a function on each event.

The function is a reducer for Stream.transform/3 which is passed the rate limit state in the accumulator:

  • t: window start time

  • limit: window threshold

  • ms: window duration in milliseconds

The function maintains a count of matching events and enables rate limiting when the threshold is exceeded for the window.

See stdio_block/2 and stdio_drop/2 for functions which can apply back pressure to or drop events from Stdio.stream!/1 or Stdio.pipe!/2.

Apply back pressure on Stdio.stream!/1 or Stdio.pipe!/2 using ratelimit/4 by blocking further reads if the rate of stdout/stderr exceeds the threshold.

Output accumulates until the process pipe buffer is full. Further writes are blocked.

Control events (process exit and termination signals) are not counted against the threshold.

Warning

When the rate limit is reached, the stream is blocked for the remainder of the window, even if the system process has exited.

To see how it works, try running:

require Logger
Stdio.stream!("while :; do date; done")
|> Stdio.Stream.ratelimit(1, 5_000, &Stdio.Stream.stdio_block/2)
|> Stream.each(fn t -> t |> inspect() |> Logger.info() end)
|> Enum.take(15)

The output from Logger will be spaced 5 seconds apart. The data from Stdio.stream!/1 is buffered:

00:11:56.410 [info]  {:stdout, "Thu Jul  7 00:11:51 EDT 2022\n"}
00:12:01.410 [info]  {:stdout, "Thu Jul  7 00:11:51 EDT 2022\n"}
00:12:06.412 [info]  {:stdout, "Thu Jul  7 00:11:51 EDT..."}

Examples

iex> Stdio.stream!("echo 1")
...> |> Stdio.Stream.ratelimit(1, 2_000, &Stdio.Stream.stdio_block/2)
...> |> Enum.to_list()
[stdout: "1\n", exit_status: 0]

Rate limit output of Stdio.stream!/1 or Stdio.pipe!/2 using ratelimit/4 by dropping events if the rate of stdout/stderr exceeds the threshold.

Control events (process exit and termination signals) are always passed through.

To see how it works, try running:

require Logger
Stdio.stream!("while :; do date; done")
|> Stdio.Stream.ratelimit(1, 5_000, &Stdio.Stream.stdio_drop/2)
|> Stream.each(fn t -> t |> inspect() |> Logger.info() end)
|> Enum.take(15)

The output from Logger will be spaced 5 seconds apart. Rate limited data from Stdio.stream!/1 is dropped:

00:17:22.058 [info]  {:stdout, "Thu Jul  7 00:17:22 EDT 2022\n"}
00:17:27.059 [info]  {:stdout, "Thu Jul  7 00:17:27 EDT 2022\n"}
00:17:32.060 [info]  {:stdout, "Thu Jul  7 00:17:32 EDT 2022\n"}
00:17:37.061 [info]  {:stdout, "Thu Jul  7 00:17:37 EDT 2022\n"}

Examples

iex> Stdio.stream!("echo 1")
...> |> Stdio.Stream.ratelimit(1, 2_000, &Stdio.Stream.stdio_drop/2)
...> |> Enum.to_list()
[stdout: "1\n", exit_status: 0]
Link to this function

stdout_to_stderr(arg, acc)

View Source
@spec stdout_to_stderr(Stdio.stdio(), term()) :: {[binary()] | :halt, term()}

Combine standard error into the standard output of Stdio.stream!/1 or Stdio.pipe!/2 and emit a list of binaries.

Exit status and termination signals are not included in the output.

Examples

iex> Stdio.stream!("echo output; echo error 1>&2; exit 1")
...> |> Stream.transform([], &Stdio.Stream.stdout_to_stderr/2)
...> |> Enum.to_list()
["output\n", "error\n"]