CollectableStreamer

A Collectable that lets you process the output of System.cmd/3 and System.shell/2 line by line, while still receiving the full output and exit code when the command finishes.

Motivation

In the Elixir standard library, this behaviour is nearly achievable:

1. To get the output and the exit code, one uses the standard System.cmd/3 and System.shell/2 functions, like this:

System.shell("ls -l")

But this does not allow us to see the output line by line as the command executes.

1. To get the output while the command is run, one can use the Mix helper Mix.Shell.cmd/3:

Mix.Shell.cmd("ls -l", [], fn x -> IO.puts(x) end)

This allows us to print the output, line by line, but it returns just the exit code. We don't get the output afterwards.

1. We can get the output from Mix.Shell.cmd/3 like this:

ExUnit.CaptureIO.with_io(fn -> Mix.Shell.cmd("ls -l", [], fn x -> IO.puts(x) end) end)

But we're back where we started because we don't see the output line by line!

To have both the ability to "see" the output as it happens and get the output and exit code afterwards, you can use CollectableStreamer.

Usage

streamer = CollectableStreamer.new(fn line -> IO.puts("Received: #{line}") end)
{result, exit_code} = System.cmd("rsync", ["-av", "foo@example.com:/source/", "/destination/"], into: streamer)

# result is a %CollectableStreamer{} — use to_string/1 to get the collected output
IO.puts("Exit code: #{exit_code}")
IO.puts("Full output:\n#{result}")

Callback examples

The callback receives each line of output as a string. Besides printing, you can use it to:

Log progress for a long-running task:

streamer = CollectableStreamer.new(fn line ->
  Logger.info("deploy: #{String.trim(line)}")
end)

Send lines to another process:

streamer = CollectableStreamer.new(fn line ->
  send(pid, {:output, line})
end)

Parse structured output:

streamer = CollectableStreamer.new(fn line ->
  case String.split(line, ",") do
    [timestamp, level, message | _] ->
      process_log_entry(timestamp, level, message)
    _ ->
      :skip
  end
end)

Disabling output collection

By default, all output lines are collected in memory so they are available after the command finishes. For long-running commands where you only need to process lines as they arrive, you can disable collection:

streamer = CollectableStreamer.new(fn line -> IO.puts(line) end, collect: false)
{_result, exit_code} = System.shell("tail -f /var/log/syslog", into: streamer)

Installation

The package can be installed by adding collectable_streamer to your list of dependencies in mix.exs:

def deps do
  [
    {:collectable_streamer, "~> 0.2.1"}
  ]
end