Tidefall 🌊

DATA RISES. THEN IT FALLS.

ETS-backed buffer for Elixir — accumulate writes, drain in periodic batches.

Tidefall is a generic, reusable buffering system inspired by the OpenTelemetry Batch Processor. It efficiently buffers arbitrary data and periodically processes it using a configurable processor function at regular intervals.

It ships with two concrete buffer implementations:

Tidefall.Queue — Ordered queue buffer (insertion-time ordered, backed by :ordered_set ETS tables). Every pushed item is buffered and drained to the processor.
Tidefall.HashMap — Coalescing key-value buffer (last-write-wins semantics, backed by :set ETS tables). Same-key writes coalesce, so only the latest value per key survives to the next tick.

Both use partitioning to reduce lock contention and double-buffering for zero-downtime processing.

📦 Installation

Add :tidefall to your list of dependencies in mix.exs:

def deps do
  [
    {:tidefall, "~> 1.0.0-rc.0"}
  ]
end

🚀 Usage

The recommended pattern is to define a dedicated buffer module and add it to your supervision tree. The module name becomes the default instance name:

defmodule MyApp.EventQueue do
  use Tidefall.Queue, otp_app: :my_app
end

defmodule MyApp.StateMap do
  use Tidefall.HashMap, otp_app: :my_app
end

# Supervision tree
children = [
  {MyApp.EventQueue, processor: &MyApp.Sink.export/1, processing_interval: 1_000},
  {MyApp.StateMap, processor: &MyApp.Sink.export/1}
]

Supervisor.start_link(children, strategy: :one_for_one)

# Queue — every item is buffered in insertion order and drained to the processor
:ok = MyApp.EventQueue.push(event)

# HashMap — same-key writes coalesce; only the latest value per key survives,
# and put_newer/4 resolves conflicts by version (newer version wins)
:ok = MyApp.StateMap.put(:user_1, %{name: "Alice"})
:ok = MyApp.StateMap.put_newer(:user_1, %{name: "Alice"}, version: 2)

The processor receives the accumulated batch on each tick and runs for its side effects (its return value is discarded):

def export(batch) do
  # Queue:   batch is [value, ...]
  # HashMap: batch is [%Tidefall.HashMap.Entry{key: k, value: v}, ...]
  Enum.each(batch, &MyApp.Sink.write/1)
end

For quick experiments or fully dynamic instances, a buffer can also be started directly with a runtime :name, e.g. Tidefall.Queue.start_link(name: :my_queue, processor: &MyApp.Sink.export/1).

See the full documentation on HexDocs for the module-based and direct-usage guides, configuration (config file and supervision tree), choosing between Queue and HashMap, telemetry events, and the complete start/runtime option reference.

🤝 Contributing

Contributions are welcome and appreciated! To report a bug, request a feature, or open a pull request, see CONTRIBUTING.md for the workflow, validation steps (mix test.ci), and commit conventions.

🙏 Acknowledgements

Tidefall began as a fork of partitioned_buffer, originally developed at Appcues. partitioned_buffer is still maintained by Appcues; Tidefall is an independent fork that has since taken a different direction. Many thanks to the Appcues team for releasing the original implementation under the MIT license.

📄 Copyright and License

Tidefall source code is licensed under the MIT License.