erlang_python

Combine Python's ML/AI ecosystem with Erlang's concurrency.

Run Python code from Erlang or Elixir with true parallelism, async/await support, and seamless integration. Build AI-powered applications that scale.

Overview

erlang_python embeds Python into the BEAM VM, letting you call Python functions, evaluate expressions, and stream from generators - all without blocking Erlang schedulers.

Three paths to parallelism:

Sub-interpreters (Python 3.12+) - Each interpreter has its own GIL
Free-threaded Python (3.13+) - No GIL at all
BEAM processes - Fan out work across lightweight Erlang processes

Key features:

Async/await - Call Python async functions, gather results, stream from async generators
Dirty NIF execution - Python runs on dirty schedulers, never blocking the BEAM
Elixir support - Works seamlessly from Elixir via the :py module
Bidirectional calls - Python can call back into registered Erlang/Elixir functions
Type conversion - Automatic conversion between Erlang and Python types
Streaming - Iterate over Python generators chunk-by-chunk
Virtual environments - Activate venvs for dependency isolation
AI/ML ready - Examples for embeddings, semantic search, RAG, and LLMs

Requirements

Erlang/OTP 27+
Python 3.12+ (3.13+ for free-threading)
C compiler (gcc, clang)

Building

rebar3 compile

Quick Start

Erlang

%% Start the application
application:ensure_all_started(erlang_python).

%% Call a Python function
{ok, 4.0} = py:call(math, sqrt, [16]).

%% With keyword arguments
{ok, Json} = py:call(json, dumps, [#{foo => bar}], #{indent => 2}).

%% Evaluate an expression
{ok, 45} = py:eval(<<"sum(range(10))">>).

%% Evaluate with local variables
{ok, 25} = py:eval(<<"x * y">>, #{x => 5, y => 5}).

%% Async calls
Ref = py:call_async(math, factorial, [100]),
{ok, Result} = py:await(Ref).

%% Streaming from generators
{ok, [0,1,4,9,16]} = py:stream_eval(<<"(x**2 for x in range(5))">>).

Elixir

# Start the application
{:ok, _} = Application.ensure_all_started(:erlang_python)

# Call Python functions
{:ok, 4.0} = :py.call(:math, :sqrt, [16])

# Evaluate expressions
{:ok, result} = :py.eval("2 + 2")

# With variables
{:ok, 100} = :py.eval("x * y", %{x: 10, y: 10})

# Call with keyword arguments
{:ok, json} = :py.call(:json, :dumps, [%{name: "Elixir"}], %{indent: 2})

Erlang/Elixir Functions Callable from Python

Erlang

%% Register a function
py:register_function(my_func, fun([X, Y]) -> X + Y end).

%% Call from Python - native import syntax (recommended)
{ok, Result} = py:exec(<<"
from erlang import my_func
result = my_func(10, 20)
">>).
%% Result = 30

%% Or use attribute-style access
{ok, 30} = py:eval(<<"erlang.my_func(10, 20)">>).

%% Legacy syntax still works
{ok, 30} = py:eval(<<"erlang.call('my_func', 10, 20)">>).

%% Unregister when done
py:unregister_function(my_func).

Elixir

# Register an Elixir function
:py.register_function(:factorial, fn [n] ->
  Enum.reduce(1..n, 1, &*/2)
end)

# Call from Python - native import syntax
{:ok, 3628800} = :py.exec("""
from erlang import factorial
result = factorial(10)
""")

# Or use attribute-style access
{:ok, 3628800} = :py.eval("erlang.factorial(10)")

Python Calling Syntax

From Python code, registered Erlang functions can be called in three ways:

# 1. Import syntax (most Pythonic)
from erlang import my_func
result = my_func(10, 20)

# 2. Attribute syntax
import erlang
result = erlang.my_func(10, 20)

# 3. Explicit call (legacy)
import erlang
result = erlang.call('my_func', 10, 20)

All three methods are equivalent. The import and attribute syntaxes provide a more natural Python experience.

Shared State Between Workers

Python workers don't share namespace state, but you can share data via the built-in state API:

From Python

from erlang import state_set, state_get, state_delete, state_keys
from erlang import state_incr, state_decr

# Store data (survives across calls, shared between workers)
state_set('my_key', {'data': [1, 2, 3], 'count': 42})

# Retrieve data
value = state_get('my_key')  # {'data': [1, 2, 3], 'count': 42}

# Atomic counters (thread-safe, great for metrics)
state_incr('requests')       # returns 1
state_incr('requests', 10)   # returns 11
state_decr('requests')       # returns 10

# List keys
keys = state_keys()  # ['my_key', 'requests', ...]

# Delete
state_delete('my_key')

From Erlang/Elixir

%% Store and fetch
py:state_store(<<"my_key">>, #{value => 42}).
{ok, #{value := 42}} = py:state_fetch(<<"my_key">>).

%% Atomic counters
1 = py:state_incr(<<"hits">>).
11 = py:state_incr(<<"hits">>, 10).
10 = py:state_decr(<<"hits">>).

%% List keys and clear
Keys = py:state_keys().
py:state_clear().

This is backed by ETS with {write_concurrency, true}, so counters are atomic and fast.

Async/Await Support

Call Python async functions without blocking:

%% Call an async function
Ref = py:async_call(aiohttp, get, [<<"https://api.example.com/data">>]),
{ok, Response} = py:async_await(Ref).

%% Gather multiple async calls concurrently
{ok, Results} = py:async_gather([
    {aiohttp, get, [<<"https://api.example.com/users">>]},
    {aiohttp, get, [<<"https://api.example.com/posts">>]},
    {aiohttp, get, [<<"https://api.example.com/comments">>]}
]).

%% Stream from async generators
{ok, Chunks} = py:async_stream(mymodule, async_generator, [args]).

Parallel Execution with Sub-interpreters

True parallelism without GIL contention using Python 3.12+ sub-interpreters:

%% Execute multiple calls in parallel across sub-interpreters
{ok, Results} = py:parallel([
    {math, factorial, [100]},
    {math, factorial, [200]},
    {math, factorial, [300]},
    {math, factorial, [400]}
]).
%% Each call runs in its own interpreter with its own GIL

Parallel Processing with BEAM Processes

Leverage Erlang's lightweight processes for massive parallelism:

%% Register parallel map function
py:register_function(parallel_map, fun([FuncName, Items]) ->
    Parent = self(),
    Refs = [begin
        Ref = make_ref(),
        spawn(fun() ->
            Result = execute(FuncName, Item),
            Parent ! {Ref, Result}
        end),
        Ref
    end || Item <- Items],
    [receive {Ref, R} -> R after 5000 -> timeout end || Ref <- Refs]
end).

%% Call from Python - processes 10 items in parallel
{ok, Results} = py:eval(
    <<"__import__('erlang').call('parallel_map', 'compute', items)">>,
    #{items => lists:seq(1, 10)}
).

Benchmark Results (from examples/erlang_concurrency.erl):

Sequential: 10 Python calls × 100ms each = 1.01 seconds
Parallel:   10 BEAM processes calling Python = 0.10 seconds

The speedup is linear with the number of items when work is I/O-bound or distributed across sub-interpreters.

Virtual Environment Support

%% Activate a venv
ok = py:activate_venv(<<"/path/to/venv">>).

%% Use packages from venv
{ok, Model} = py:call(sentence_transformers, 'SentenceTransformer', [<<"all-MiniLM-L6-v2">>]).

%% Deactivate when done
ok = py:deactivate_venv().

Examples

The examples/ directory contains runnable demonstrations:

Semantic Search

# Setup
python3 -m venv /tmp/ai-venv
/tmp/ai-venv/bin/pip install sentence-transformers numpy

# Run
escript examples/semantic_search.erl

RAG (Retrieval-Augmented Generation)

# Setup (also install Ollama and pull a model)
/tmp/ai-venv/bin/pip install sentence-transformers numpy requests
ollama pull llama3.2

# Run
escript examples/rag_example.erl

AI Chat

escript examples/ai_chat.erl

Erlang Concurrency from Python

# Demonstrates 10x speedup with BEAM processes
escript examples/erlang_concurrency.erl

Elixir Integration

elixir --erl "-pa _build/default/lib/erlang_python/ebin" examples/elixir_example.exs

API Reference

Function Calls

{ok, Result} = py:call(Module, Function, Args).
{ok, Result} = py:call(Module, Function, Args, KwArgs).
{ok, Result} = py:call(Module, Function, Args, KwArgs, Timeout).

%% Async
Ref = py:call_async(Module, Function, Args).
{ok, Result} = py:await(Ref).
{ok, Result} = py:await(Ref, Timeout).

Expression Evaluation

{ok, 42} = py:eval(<<"21 * 2">>).
{ok, 100} = py:eval(<<"x * y">>, #{x => 10, y => 10}).
{ok, Result} = py:eval(Expression, Locals, Timeout).

Streaming

{ok, Chunks} = py:stream(Module, GeneratorFunc, Args).
{ok, [0,1,4,9,16]} = py:stream_eval(<<"(x**2 for x in range(5))">>).

Callbacks

py:register_function(Name, fun([Args]) -> Result end).
py:register_function(Name, Module, Function).
py:unregister_function(Name).

Memory and GC

{ok, Stats} = py:memory_stats().
{ok, Collected} = py:gc().
ok = py:tracemalloc_start().
ok = py:tracemalloc_stop().

Type Mappings

Erlang to Python

Erlang	Python
`integer()`	`int`
`float()`	`float`
`binary()`	`str`
`atom()`	`str`
`true` / `false`	`True` / `False`
`none` / `nil`	`None`
`list()`	`list`
`tuple()`	`tuple`
`map()`	`dict`

Python to Erlang

Python	Erlang
`int`	`integer()`
`float`	`float()`
`str`	`binary()`
`bytes`	`binary()`
`True` / `False`	`true` / `false`
`None`	`none`
`list`	`list()`
`tuple`	`tuple()`
`dict`	`map()`

Configuration

%% sys.config
[
  {erlang_python, [
    {num_workers, 4},           %% Python worker pool size
    {max_concurrent, 17},       %% Max concurrent operations (default: schedulers * 2 + 1)
    {num_executors, 4}          %% Executor threads (multi-executor mode)
  ]}
].

Execution Modes

The library auto-detects the best execution mode:

Mode	Python Version	Parallelism
Free-threaded	3.13+ (nogil)	True parallel, no GIL
Sub-interpreter	3.12+	Per-interpreter GIL
Multi-executor	Any	GIL contention

Check current mode:

py:execution_mode().  %% => free_threaded | subinterp | multi_executor

Error Handling

{error, {'NameError', "name 'x' is not defined"}} = py:eval(<<"x">>).
{error, {'ZeroDivisionError', "division by zero"}} = py:eval(<<"1/0">>).
{error, timeout} = py:eval(<<"sum(range(10**9))">>, #{}, 100).

Documentation

License

Apache-2.0