# erlang_python

**Combine Python's ML/AI ecosystem with Erlang's concurrency.**

Run Python code from Erlang or Elixir with true parallelism, async/await support,
and seamless integration. Build AI-powered applications that scale.

## Overview

erlang_python embeds Python into the BEAM VM, letting you call Python functions,
evaluate expressions, and stream from generators - all without blocking Erlang
schedulers.

**Three paths to parallelism:**
- **Sub-interpreters** (Python 3.12+) - Each interpreter has its own GIL
- **Free-threaded Python** (3.13+) - No GIL at all
- **BEAM processes** - Fan out work across lightweight Erlang processes

Key features:
- **Async/await** - Call Python async functions, gather results, stream from async generators
- **Dirty NIF execution** - Python runs on dirty schedulers, never blocking the BEAM
- **Elixir support** - Works seamlessly from Elixir via the `:py` module
- **Bidirectional calls** - Python can call back into registered Erlang/Elixir functions
- **Type conversion** - Automatic conversion between Erlang and Python types
- **Streaming** - Iterate over Python generators chunk-by-chunk
- **Virtual environments** - Activate venvs for dependency isolation
- **AI/ML ready** - Examples for embeddings, semantic search, RAG, and LLMs

## Requirements

- Erlang/OTP 27+
- Python 3.12+ (3.13+ for free-threading)
- C compiler (gcc, clang)

## Building

```bash
rebar3 compile
```

## Quick Start

### Erlang

```erlang
%% Start the application
application:ensure_all_started(erlang_python).

%% Call a Python function
{ok, 4.0} = py:call(math, sqrt, [16]).

%% With keyword arguments
{ok, Json} = py:call(json, dumps, [#{foo => bar}], #{indent => 2}).

%% Evaluate an expression
{ok, 45} = py:eval(<<"sum(range(10))">>).

%% Evaluate with local variables
{ok, 25} = py:eval(<<"x * y">>, #{x => 5, y => 5}).

%% Async calls
Ref = py:call_async(math, factorial, [100]),
{ok, Result} = py:await(Ref).

%% Streaming from generators
{ok, [0,1,4,9,16]} = py:stream_eval(<<"(x**2 for x in range(5))">>).
```

### Elixir

```elixir
# Start the application
{:ok, _} = Application.ensure_all_started(:erlang_python)

# Call Python functions
{:ok, 4.0} = :py.call(:math, :sqrt, [16])

# Evaluate expressions
{:ok, result} = :py.eval("2 + 2")

# With variables
{:ok, 100} = :py.eval("x * y", %{x: 10, y: 10})

# Call with keyword arguments
{:ok, json} = :py.call(:json, :dumps, [%{name: "Elixir"}], %{indent: 2})
```

## Erlang/Elixir Functions Callable from Python

Register Erlang or Elixir functions that Python code can call back into:

### Erlang

```erlang
%% Register a function
py:register_function(my_func, fun([X, Y]) -> X + Y end).

%% Call from Python
{ok, Result} = py:eval(<<"__import__('erlang').call('my_func', 10, 20)">>).
%% Result = 30

%% Unregister when done
py:unregister_function(my_func).
```

### Elixir

```elixir
# Register an Elixir function
:py.register_function(:factorial, fn [n] ->
  Enum.reduce(1..n, 1, &*/2)
end)

# Call from Python
{:ok, 3628800} = :py.eval("__import__('erlang').call('factorial', 10)")
```

## Async/Await Support

Call Python async functions without blocking:

```erlang
%% Call an async function
Ref = py:async_call(aiohttp, get, [<<"https://api.example.com/data">>]),
{ok, Response} = py:async_await(Ref).

%% Gather multiple async calls concurrently
{ok, Results} = py:async_gather([
    {aiohttp, get, [<<"https://api.example.com/users">>]},
    {aiohttp, get, [<<"https://api.example.com/posts">>]},
    {aiohttp, get, [<<"https://api.example.com/comments">>]}
]).

%% Stream from async generators
{ok, Chunks} = py:async_stream(mymodule, async_generator, [args]).
```

## Parallel Execution with Sub-interpreters

True parallelism without GIL contention using Python 3.12+ sub-interpreters:

```erlang
%% Execute multiple calls in parallel across sub-interpreters
{ok, Results} = py:parallel([
    {math, factorial, [100]},
    {math, factorial, [200]},
    {math, factorial, [300]},
    {math, factorial, [400]}
]).
%% Each call runs in its own interpreter with its own GIL
```

## Parallel Processing with BEAM Processes

Leverage Erlang's lightweight processes for massive parallelism:

```erlang
%% Register parallel map function
py:register_function(parallel_map, fun([FuncName, Items]) ->
    Parent = self(),
    Refs = [begin
        Ref = make_ref(),
        spawn(fun() ->
            Result = execute(FuncName, Item),
            Parent ! {Ref, Result}
        end),
        Ref
    end || Item <- Items],
    [receive {Ref, R} -> R after 5000 -> timeout end || Ref <- Refs]
end).

%% Call from Python - processes 10 items in parallel
{ok, Results} = py:eval(
    <<"__import__('erlang').call('parallel_map', 'compute', items)">>,
    #{items => lists:seq(1, 10)}
).
```

**Benchmark Results** (from `examples/erlang_concurrency.erl`):
```
Sequential: 10 Python calls × 100ms each = 1.01 seconds
Parallel:   10 BEAM processes calling Python = 0.10 seconds
```

The speedup is linear with the number of items when work is I/O-bound or
distributed across sub-interpreters.

## Virtual Environment Support

```erlang
%% Activate a venv
ok = py:activate_venv(<<"/path/to/venv">>).

%% Use packages from venv
{ok, Model} = py:call(sentence_transformers, 'SentenceTransformer', [<<"all-MiniLM-L6-v2">>]).

%% Deactivate when done
ok = py:deactivate_venv().
```

## Examples

The `examples/` directory contains runnable demonstrations:

### Semantic Search
```bash
# Setup
python3 -m venv /tmp/ai-venv
/tmp/ai-venv/bin/pip install sentence-transformers numpy

# Run
escript examples/semantic_search.erl
```

### RAG (Retrieval-Augmented Generation)
```bash
# Setup (also install Ollama and pull a model)
/tmp/ai-venv/bin/pip install sentence-transformers numpy requests
ollama pull llama3.2

# Run
escript examples/rag_example.erl
```

### AI Chat
```bash
escript examples/ai_chat.erl
```

### Erlang Concurrency from Python
```bash
# Demonstrates 10x speedup with BEAM processes
escript examples/erlang_concurrency.erl
```

### Elixir Integration
```bash
elixir --erl "-pa _build/default/lib/erlang_python/ebin" examples/elixir_example.exs
```

## API Reference

### Function Calls

```erlang
{ok, Result} = py:call(Module, Function, Args).
{ok, Result} = py:call(Module, Function, Args, KwArgs).
{ok, Result} = py:call(Module, Function, Args, KwArgs, Timeout).

%% Async
Ref = py:call_async(Module, Function, Args).
{ok, Result} = py:await(Ref).
{ok, Result} = py:await(Ref, Timeout).
```

### Expression Evaluation

```erlang
{ok, 42} = py:eval(<<"21 * 2">>).
{ok, 100} = py:eval(<<"x * y">>, #{x => 10, y => 10}).
{ok, Result} = py:eval(Expression, Locals, Timeout).
```

### Streaming

```erlang
{ok, Chunks} = py:stream(Module, GeneratorFunc, Args).
{ok, [0,1,4,9,16]} = py:stream_eval(<<"(x**2 for x in range(5))">>).
```

### Callbacks

```erlang
py:register_function(Name, fun([Args]) -> Result end).
py:register_function(Name, Module, Function).
py:unregister_function(Name).
```

### Memory and GC

```erlang
{ok, Stats} = py:memory_stats().
{ok, Collected} = py:gc().
ok = py:tracemalloc_start().
ok = py:tracemalloc_stop().
```

## Type Mappings

### Erlang to Python

| Erlang | Python |
|--------|--------|
| `integer()` | `int` |
| `float()` | `float` |
| `binary()` | `str` |
| `atom()` | `str` |
| `true` / `false` | `True` / `False` |
| `none` / `nil` | `None` |
| `list()` | `list` |
| `tuple()` | `tuple` |
| `map()` | `dict` |

### Python to Erlang

| Python | Erlang |
|--------|--------|
| `int` | `integer()` |
| `float` | `float()` |
| `str` | `binary()` |
| `bytes` | `binary()` |
| `True` / `False` | `true` / `false` |
| `None` | `none` |
| `list` | `list()` |
| `tuple` | `tuple()` |
| `dict` | `map()` |

## Configuration

```erlang
%% sys.config
[
  {erlang_python, [
    {num_workers, 4},           %% Python worker pool size
    {max_concurrent, 17},       %% Max concurrent operations (default: schedulers * 2 + 1)
    {num_executors, 4}          %% Executor threads (multi-executor mode)
  ]}
].
```

## Execution Modes

The library auto-detects the best execution mode:

| Mode | Python Version | Parallelism |
|------|----------------|-------------|
| Free-threaded | 3.13+ (nogil) | True parallel, no GIL |
| Sub-interpreter | 3.12+ | Per-interpreter GIL |
| Multi-executor | Any | GIL contention |

Check current mode:
```erlang
py:execution_mode().  %% => free_threaded | subinterp | multi_executor
```

## Error Handling

```erlang
{error, {'NameError', "name 'x' is not defined"}} = py:eval(<<"x">>).
{error, {'ZeroDivisionError', "division by zero"}} = py:eval(<<"1/0">>).
{error, timeout} = py:eval(<<"sum(range(10**9))">>, #{}, 100).
```

## Documentation

- [Getting Started](docs/getting-started.md)
- [AI Integration Guide](docs/ai-integration.md)
- [Type Conversion](docs/type-conversion.md)
- [Scalability](docs/scalability.md)
- [Streaming](docs/streaming.md)

## License

Apache-2.0