LiveCompile

LiveCompile provides lightning-fast ad-hoc data transformations in Livebook and IEx by leveraging dynamic JIT (Just-In-Time) compilation directly into the Erlang VM (BEAM) memory.

The Problem

When working with larger datasets in interactive environments like Livebook or the IEx shell, data manipulation pipelines often become incredibly slow. For example, running Enum.map/2 with an anonymous function that does non-trivial transformation can take seconds or even minutes.

This happens because Livebook and IEx evaluate expressions step-by-step using an interpreter (:erl_eval). Anonymous lambdas declared inside cells are interpreted on every single iteration instead of being compiled into bytecode, bypassing BEAM's powerful compiler optimizations including JIT compilation.

The Solution

LiveCompile solves this by capturing your pipeline's Abstract Syntax Tree (AST) using macros, automatically grabbing all available variables from the caller's environment (Macro.Env.vars/1), and instantly compiling the entire block as a native module using Module.create/3.

The compilation overhead is typically around 10 millisecond, but the execution runs at full production-grade compiled speed, easily resulting in 10x to 100x performance improvements during interactive data analysis.

Installation

Add live_compile to your dependencies in Livebook's setup cell or your mix.exs:

Mix.install([
  {:live_compile, "~> 0.1.0"}
])

Note: Requires Elixir 1.15+ as it relies on Macro.Env.vars/1.

Usage

Import LiveCompile at the beginning of your data analysis cell to unlock three flexible ways of boosting your pipeline performance using the e macro or surgical ~> pipe operator.

1. Block Macro (e do ... end)

Wrap a full multi-line data pipeline. The entire block gets compiled into a single temporary native module. This offers the best performance since BEAM can optimize inter-function transitions as a single unit.

import LiveCompile

diffs = e do
  msgs
  |> Enum.filter(&Map.has_key?(&1.value, :end))
  |> Enum.map(&%{ts: &1.timestamp, diff: &1.value.end - &1.timestamp})
end

2. Surgical Pipeline Operator (~>)

If you only want to compile a single heavy transformation line within a standard Elixir pipeline, swap |> for ~>. Everything before it remains interpreted, but the right-hand expression gets isolated and compiled natively.

import LiveCompile

diffs =
  msgs
  |> Enum.filter(& &1.timestamp > 1000)   # Interpreted
  ~> Enum.map(& %{&1 | value: :done})     # COMPILED
  |> Enum.take(5)                         # Interpreted

Tip: You can chain multiple ~> operators sequentially, or wrap a sub-pipeline in parentheses ~> (A |> B) to group them into a single compiled module.

3. Inline Macro Helper (|> e(...))

An alternative syntax to the ~> operator designed to fit natively into standard pipeline visual styles. It acts identically to the operator under the hood.

import LiveCompile

diffs =
  msgs
  |> e(Enum.filter(&Map.has_key?(&1.value, :end)))
  |> Enum.take(5)

Micro-benchmarking Included

Every time a LiveCompile macro executes, it measures the exact time spent on module generation. It automatically prints a clear blueprint right under your Livebook cell showing the line number and compile time:

5:compiled in 8.42 ms

This allows you to verify instantly whether the compilation JIT step was worth it (which it always is for larger collections!).

How Error Reporting and Closures Work

• Automatic Closures: You do not need to manually pass variable bindings. The macro safely inspects your local scope and brings external datasets (msgs, configuration flags, keys, etc.) right inside the isolated compiled module.
• Accurate Stacktraces: Thanks to Macro.Env.location/1 enforcement during module spawning, any runtime exceptions or crashes (e.g. failed pattern matches) will correctly highlight the exact line inside your original Livebook cell, rather than blaming an anonymous background module.

License

LiveCompile is released under the MIT License.