View Source Stdio.Stream (stdio v0.4.4)
Stream standard I/O from system processes
Summary
Functions
Rate limits a stream by discarding elements exceeding a threshold for the remainder of the window.
Rate limits a stream by calling a function on each event.
Apply back pressure on Stdio.stream!/1
or Stdio.pipe!/2
using
ratelimit/4
by blocking further reads if the rate of stdout/stderr
exceeds the threshold.
Rate limit output of Stdio.stream!/1
or Stdio.pipe!/2
using
ratelimit/4
by dropping events if the rate of stdout/stderr exceeds
the threshold.
Combine standard error into the standard output of Stdio.stream!/1
or Stdio.pipe!/2
and emit a list of binaries.
Types
@type t() :: %Stdio.Stream{ flush_timeout: 0 | :infinity, onexit: (Stdio.ProcessTree.t() -> boolean()), process: Stdio.ProcessTree.t() | nil, status: :running | :flush | :flushing, stream_pid: pid() | nil }
Functions
@spec ratelimit(Enumerable.t(), pos_integer(), pos_integer()) :: Enumerable.t()
Rate limits a stream by discarding elements exceeding a threshold for the remainder of the window.
ratelimit/3
works with any Enumerable.t
. For working with streams
generated by Stdio
, see ratelimit/4
.
Examples
iex> Stream.unfold(1_000, fn 0 -> nil; n -> {n, n-1} end)
...> |> Stdio.Stream.ratelimit(2, 10_000)
...> |> Enum.to_list()
[1000, 999]
@spec ratelimit( Enumerable.t(), pos_integer(), pos_integer(), (term(), acc -> {Enumerable.t(), acc} | {:halt, term()}) ) :: Enumerable.t() when acc: %{t: integer(), limit: non_neg_integer(), ms: non_neg_integer()}
Rate limits a stream by calling a function on each event.
The function is a reducer for Stream.transform/3
which is passed
the rate limit state in the accumulator:
t: window start time
limit: window threshold
ms: window duration in milliseconds
The function maintains a count of matching events and enables rate limiting when the threshold is exceeded for the window.
See stdio_block/2
and stdio_drop/2
for functions which can apply
back pressure to or drop events from Stdio.stream!/1
or Stdio.pipe!/2
.
Apply back pressure on Stdio.stream!/1
or Stdio.pipe!/2
using
ratelimit/4
by blocking further reads if the rate of stdout/stderr
exceeds the threshold.
Output accumulates until the process pipe buffer is full. Further writes are blocked.
Control events (process exit and termination signals) are not counted against the threshold.
Warning
When the rate limit is reached, the stream is blocked for the remainder of the window, even if the system process has exited.
To see how it works, try running:
require Logger
Stdio.stream!("while :; do date; done")
|> Stdio.Stream.ratelimit(1, 5_000, &Stdio.Stream.stdio_block/2)
|> Stream.each(fn t -> t |> inspect() |> Logger.info() end)
|> Enum.take(15)
The output from Logger will be spaced 5 seconds apart. The data from
Stdio.stream!/1
is buffered:
00:11:56.410 [info] {:stdout, "Thu Jul 7 00:11:51 EDT 2022\n"}
00:12:01.410 [info] {:stdout, "Thu Jul 7 00:11:51 EDT 2022\n"}
00:12:06.412 [info] {:stdout, "Thu Jul 7 00:11:51 EDT..."}
Examples
iex> Stdio.stream!("echo 1")
...> |> Stdio.Stream.ratelimit(1, 2_000, &Stdio.Stream.stdio_block/2)
...> |> Enum.to_list()
[stdout: "1\n", exit_status: 0]
Rate limit output of Stdio.stream!/1
or Stdio.pipe!/2
using
ratelimit/4
by dropping events if the rate of stdout/stderr exceeds
the threshold.
Control events (process exit and termination signals) are always passed through.
To see how it works, try running:
require Logger
Stdio.stream!("while :; do date; done")
|> Stdio.Stream.ratelimit(1, 5_000, &Stdio.Stream.stdio_drop/2)
|> Stream.each(fn t -> t |> inspect() |> Logger.info() end)
|> Enum.take(15)
The output from Logger will be spaced 5 seconds apart. Rate limited
data from Stdio.stream!/1
is dropped:
00:17:22.058 [info] {:stdout, "Thu Jul 7 00:17:22 EDT 2022\n"}
00:17:27.059 [info] {:stdout, "Thu Jul 7 00:17:27 EDT 2022\n"}
00:17:32.060 [info] {:stdout, "Thu Jul 7 00:17:32 EDT 2022\n"}
00:17:37.061 [info] {:stdout, "Thu Jul 7 00:17:37 EDT 2022\n"}
Examples
iex> Stdio.stream!("echo 1")
...> |> Stdio.Stream.ratelimit(1, 2_000, &Stdio.Stream.stdio_drop/2)
...> |> Enum.to_list()
[stdout: "1\n", exit_status: 0]
@spec stdout_to_stderr(Stdio.stdio(), term()) :: {[binary()] | :halt, term()}
Combine standard error into the standard output of Stdio.stream!/1
or Stdio.pipe!/2
and emit a list of binaries.
Exit status and termination signals are not included in the output.
Examples
iex> Stdio.stream!("echo output; echo error 1>&2; exit 1")
...> |> Stream.transform([], &Stdio.Stream.stdout_to_stderr/2)
...> |> Enum.to_list()
["output\n", "error\n"]