Traveller

Traveller provides a easy, stream-based traversal of a database table. This library is well-suited for backfills, migrations etc. It can handle massive tables because of the lazy enumeration.

Function approach

With cursors

The most basic use case is:

MyRepo
|> Traveller.start_stream(MySchema)
|> Enum.map(fn record_batch ->
  # your logic here
 end)

You can also handle more advanced use cases. Let's say you have a users table and you want to iterate over all users, sorting by first name in reverse alphabetical order, then last name in reverse order, then id. But you want to start after "Z" and stop before "A". Also, you want a small batch size. You can do something like:

MyRepo
|> Traveller.start_stream(
  MySchema,
  start_after: "Z",
  stop_before: "A",
  cursor: [desc: :first_name, desc: :last_name, asc: :id],
  chunk_size: 50
)
|> Enum.map(fn record_batch ->
  # your logic here
 end)

With offset

Assuming you have the correct indexes cursor-based iteration is normally much more efficient. But if you want to do an offset based traversal, for whatever reason that is possible too:

MyRepo
|> Traveller.start_stream(
  MySchema,
  mode: :offset,
  order_by: :id
)
|> Enum.map(fn record_batch ->
  # your logic here
 end)

Module approach

You can also use it to bake certain options into a module. These can then be overridden later:

defmodule YourModule do
  use Traveller,
    repo: Repo,
    schema: Person,
    cursor: :first_name,
    chunk_size: 100
end

Enum.each(YourModule.start_stream(chunk_size: 50), fn batch ->
  # do something
end)

Installation

If available in Hex, the package can be installed by adding traveller to your list of dependencies in mix.exs:

def deps do
  [
    {:traveller, "~> 0.1.0"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/traveller.