How to make Elixir’s File.Stream! work with compressed files

March 31st, 2021

From time to time I need to work with largish gzip compressed files in Elixir. Rather than uncompress these externally it’s possible to stream from the compressed files by passing in some options to the File.stream!/3 call.

This looks like:

defmodule Streamy do
  def stream(some_file_path) do
    some_file_path
    |> File.stream!([{:read_ahead, 100_000}, :compressed])
    |> Stream.map(&IO.inspect/1)
    |> Stream.run()
  end
end

You don’t need the {:read_ahead, 100_000} option but it’s worth knowing it in case you want to tune the IO.

This works because File.stream!/3 wraps Erlang’s file:open/2 call and :compressed is one of the options you can pass to that.

It’s worth having a skim over the Erlang docs for other options that might be useful.