# Process-Bound Python Environments
*Since version 2.2.0*
## Philosophy
In Erlang, processes are the fundamental unit of isolation. Each process has its own heap, mailbox, and lifecycle. When a process crashes, it takes its state with it and can be restarted clean by a supervisor.
erlang_python extends this philosophy to Python: **each Erlang process gets its own isolated Python environment**. Variables, imports, and objects defined in one process are invisible to others, even when using the same Python context.
This design enables:
- **Clean restarts**: Resetting Python state = terminating the Erlang process
- **Fault isolation**: A corrupted Python state crashes only its owning process
- **Supervision**: Standard OTP supervisors can manage Python-backed actors
- **Actor model**: Build stateful Python services that behave like gen_servers
## How It Works
When you call `py:exec/eval/call`, the library:
1. Looks up a process-local environment keyed by `{ContextPid, InterpreterId}`
2. Creates one if it doesn't exist (a Python `dict` inside the interpreter)
3. Uses that dict as the namespace for execution
4. Cleans up automatically when the Erlang process exits
```erlang
%% Process A
spawn(fun() ->
Ctx = py:context(1),
ok = py:exec(Ctx, <<"state = 'hello'">>),
{ok, <<"hello">>} = py:eval(Ctx, <<"state">>)
end).
%% Process B - same context, isolated state
spawn(fun() ->
Ctx = py:context(1),
%% state is undefined here - different process
{error, _} = py:eval(Ctx, <<"state">>)
end).
```
## OWN_GIL Mode
OWN_GIL contexts (Python 3.12+) provide true parallel execution with dedicated pthreads. Process-bound environments work with OWN_GIL, allowing multiple Erlang processes to share a single OWN_GIL context while maintaining isolated Python namespaces.
### Explicit Environment Creation
For OWN_GIL contexts, you can explicitly create and manage environments:
```erlang
%% Create an OWN_GIL context
{ok, Ctx} = py_context:start_link(1, owngil),
%% Create a process-local environment
{ok, Env} = py_context:create_local_env(Ctx),
%% Get the NIF reference for low-level operations
CtxRef = py_context:get_nif_ref(Ctx),
%% Execute code in the isolated environment
ok = py_nif:context_exec(CtxRef, <<"
class MyService:
def __init__(self):
self.counter = 0
def increment(self):
self.counter += 1
return self.counter
service = MyService()
">>, Env),
%% Call functions in the environment
{ok, 1} = py_nif:context_eval(CtxRef, <<"service.increment()">>, #{}, Env),
{ok, 2} = py_nif:context_eval(CtxRef, <<"service.increment()">>, #{}, Env).
```
### Sharing Context, Isolating State
Multiple Erlang processes can share an OWN_GIL context while maintaining isolated namespaces:
```erlang
%% Shared OWN_GIL context
{ok, Ctx} = py_context:start_link(1, owngil),
CtxRef = py_context:get_nif_ref(Ctx),
%% Process A - its own namespace
spawn(fun() ->
{ok, EnvA} = py_context:create_local_env(Ctx),
ok = py_nif:context_exec(CtxRef, <<"x = 'from A'">>, EnvA),
{ok, <<"from A">>} = py_nif:context_eval(CtxRef, <<"x">>, #{}, EnvA)
end),
%% Process B - separate namespace, same context
spawn(fun() ->
{ok, EnvB} = py_context:create_local_env(Ctx),
ok = py_nif:context_exec(CtxRef, <<"x = 'from B'">>, EnvB),
{ok, <<"from B">>} = py_nif:context_eval(CtxRef, <<"x">>, #{}, EnvB)
end).
%% Both execute in parallel on the same OWN_GIL thread, but with isolated state
```
### When to Use Explicit vs Implicit Environments
| Approach | API | Use Case |
|----------|-----|----------|
| **Implicit** | `py:exec/eval/call` | Simple cases, automatic management |
| **Explicit** | `create_local_env` + `py_nif:context_*` | OWN_GIL, fine-grained control, multiple envs per process |
**Use implicit (py:exec)** when:
- Using worker or subinterp modes
- One environment per process is sufficient
- You want automatic lifecycle management
**Use explicit (create_local_env)** when:
- Using OWN_GIL mode for parallel execution
- Need multiple environments in a single process
- Want to pass environments between processes
- Need direct NIF-level control
## Event Loop Environments
The event loop API also supports per-process namespaces. Each Erlang process gets an isolated namespace within the event loop, allowing you to define functions and state that persist across async task calls.
### Defining Functions for Async Tasks
```erlang
%% Get the event loop reference
{ok, Loop} = py_event_loop:get_loop(),
LoopRef = py_event_loop:get_nif_ref(Loop),
%% Define a function in this process's namespace
ok = py_nif:event_loop_exec(LoopRef, <<"
import asyncio
async def my_async_function(x):
await asyncio.sleep(0.1)
return x * 2
counter = 0
async def increment_and_get():
global counter
counter += 1
return counter
">>),
%% Call the function via create_task - uses __main__ module
{ok, Ref} = py_event_loop:create_task(Loop, '__main__', my_async_function, [21]),
{ok, 42} = py_event_loop:await(Ref),
%% State persists across calls
{ok, Ref1} = py_event_loop:create_task(Loop, '__main__', increment_and_get, []),
{ok, 1} = py_event_loop:await(Ref1),
{ok, Ref2} = py_event_loop:create_task(Loop, '__main__', increment_and_get, []),
{ok, 2} = py_event_loop:await(Ref2).
```
### Evaluating Expressions
```erlang
%% Evaluate expressions in the process's namespace
{ok, 42} = py_nif:event_loop_eval(LoopRef, <<"21 * 2">>),
%% Access variables defined via exec
ok = py_nif:event_loop_exec(LoopRef, <<"result = 'computed'">>),
{ok, <<"computed">>} = py_nif:event_loop_eval(LoopRef, <<"result">>).
```
### Process Isolation
Different Erlang processes have isolated event loop namespaces:
```erlang
{ok, Loop} = py_event_loop:get_loop(),
LoopRef = py_event_loop:get_nif_ref(Loop),
%% Process A defines x
spawn(fun() ->
ok = py_nif:event_loop_exec(LoopRef, <<"x = 'A'">>),
{ok, <<"A">>} = py_nif:event_loop_eval(LoopRef, <<"x">>)
end),
%% Process B has its own x
spawn(fun() ->
ok = py_nif:event_loop_exec(LoopRef, <<"x = 'B'">>),
{ok, <<"B">>} = py_nif:event_loop_eval(LoopRef, <<"x">>)
end).
```
### Cleanup
Event loop namespaces are automatically cleaned up when the Erlang process exits. The event loop monitors each process that creates a namespace and removes it on process termination.
### Automatic Env Reuse with py:exec
Functions defined via `py:exec(Ctx, Code)` can be called directly using the async task API (`py_event_loop:run/3,4`, `create_task/3,4`, `spawn_task/3,4`). The process-local environment is automatically detected and used for function lookup.
```erlang
%% Create a context and define an async function
Ctx = py:context(1),
ok = py:exec(Ctx, <<"
import asyncio
async def process_data(items):
results = []
for item in items:
await asyncio.sleep(0.001)
results.append(item * 2)
return results
">>),
%% Call it directly - env is reused automatically
{ok, [2,4,6]} = py_event_loop:run('__main__', process_data, [[1,2,3]]).
%% Also works with create_task and spawn_task
Ref = py_event_loop:create_task('__main__', process_data, [[4,5,6]]),
{ok, [8,10,12]} = py_event_loop:await(Ref).
%% Fire-and-forget
ok = py_event_loop:spawn_task('__main__', process_data, [[7,8,9]]).
```
This eliminates the need to manually pass environment references when calling functions defined in the process-local namespace.
## Building Python Actors
The process-bound model enables a pattern we call "Python actors" - Erlang processes that encapsulate Python state and expose it through message passing.
### Basic Actor Pattern
```erlang
-module(py_counter).
-behaviour(gen_server).
-export([start_link/0, increment/1, decrement/1, get/1]).
-export([init/1, handle_call/3, handle_cast/2]).
start_link() ->
gen_server:start_link(?MODULE, [], []).
increment(Pid) -> gen_server:call(Pid, increment).
decrement(Pid) -> gen_server:call(Pid, decrement).
get(Pid) -> gen_server:call(Pid, get).
init([]) ->
Ctx = py:context(),
ok = py:exec(Ctx, <<"
class Counter:
def __init__(self):
self.value = 0
def increment(self):
self.value += 1
return self.value
def decrement(self):
self.value -= 1
return self.value
def get(self):
return self.value
counter = Counter()
">>),
{ok, #{ctx => Ctx}}.
handle_call(increment, _From, #{ctx := Ctx} = State) ->
{ok, Value} = py:eval(Ctx, <<"counter.increment()">>),
{reply, Value, State};
handle_call(decrement, _From, #{ctx := Ctx} = State) ->
{ok, Value} = py:eval(Ctx, <<"counter.decrement()">>),
{reply, Value, State};
handle_call(get, _From, #{ctx := Ctx} = State) ->
{ok, Value} = py:eval(Ctx, <<"counter.get()">>),
{reply, Value, State}.
handle_cast(_Msg, State) ->
{noreply, State}.
```
Usage:
```erlang
{ok, Counter} = py_counter:start_link(),
1 = py_counter:increment(Counter),
2 = py_counter:increment(Counter),
1 = py_counter:decrement(Counter),
1 = py_counter:get(Counter).
```
### Reset via Process Termination
Following Erlang's "let it crash" philosophy, resetting Python state is simple:
```erlang
%% Supervise the Python actor
init([]) ->
Children = [
#{id => py_worker,
start => {py_worker, start_link, []},
restart => permanent}
],
{ok, {#{strategy => one_for_one}, Children}}.
%% To reset: just terminate and let supervisor restart
reset_worker(Sup) ->
ok = supervisor:terminate_child(Sup, py_worker),
{ok, _} = supervisor:restart_child(Sup, py_worker).
```
No need to manually clear variables, reload modules, or reset interpreter state. The new process starts with a fresh Python environment.
### Stateful ML Pipeline
```erlang
-module(ml_predictor).
-behaviour(gen_server).
-export([start_link/1, predict/2]).
-export([init/1, handle_call/3, terminate/2]).
start_link(ModelPath) ->
gen_server:start_link(?MODULE, ModelPath, []).
predict(Pid, Features) ->
gen_server:call(Pid, {predict, Features}).
init(ModelPath) ->
Ctx = py:context(),
%% Define functions and load model - stored in process-bound environment
ok = py:exec(Ctx, <<"
import pickle
_model = None
def load_model(path):
global _model
with open(path, 'rb') as f:
_model = pickle.load(f)
return True
def predict(features):
return _model.predict([features]).tolist()[0]
">>),
%% Load model - it's stored in _model global within this process's env
{ok, true} = py:call(Ctx, '__main__', load_model, [ModelPath]),
{ok, #{ctx => Ctx}}.
handle_call({predict, Features}, _From, #{ctx := Ctx} = State) ->
{ok, Result} = py:call(Ctx, '__main__', predict, [Features]),
{reply, {ok, Result}, State}.
terminate(_Reason, _State) ->
%% Python environment automatically cleaned up
ok.
```
## Advantages
| Aspect | Benefit |
|--------|---------|
| **Isolation** | Processes cannot interfere with each other's Python state |
| **Cleanup** | No resource leaks - process death = environment cleanup |
| **Restart** | Fresh state by terminating process (no manual reset logic) |
| **Supervision** | OTP supervisors manage Python actors like any other process |
| **Debugging** | Process dictionary inspection shows environment reference |
| **Memory** | Each process's Python memory counted separately |
## Trade-offs
| Aspect | Consideration |
|--------|---------------|
| **Memory overhead** | Each process has separate Python dict; no sharing |
| **Startup cost** | Environment created on first call per process |
| **No shared state** | State sharing requires explicit message passing or ETS |
| **Module caching** | Imported modules cached at interpreter level, not per-process |
## When to Use
**Good fit:**
- Stateful services (sessions, connections, workflows)
- Actor-style Python components
- Isolated workers that may need reset
- Per-request processing with state accumulation
- Supervised Python services
**Consider alternatives when:**
- Sharing state between many processes (use ETS or message passing)
- State must survive process restarts (use external storage)
- Memory is constrained (many processes = many environments)
- Truly stateless operations (environment overhead unnecessary)
## Comparison with Other Models
### vs. Global Interpreter State
Traditional Python embedding shares state globally. Any code can modify any variable. Isolation requires explicit namespace management.
With process-bound environments:
```erlang
%% Each process is automatically isolated
spawn(fun() -> py:exec(Ctx, <<"x = 1">>) end),
spawn(fun() -> py:exec(Ctx, <<"x = 2">>) end).
%% No conflict - different environments
```
### vs. Multiple Interpreters
Some systems create separate Python interpreters per "session". This provides isolation but:
- High memory cost per interpreter
- GIL contention in multi-interpreter setups
- Complex lifecycle management
Process-bound environments use a single interpreter (or subinterpreter pool) but isolate at the namespace level - lightweight and efficient.
### vs. Stateless Lambda-Style
Some systems treat Python as pure functions with no state between calls:
```erlang
%% Stateless style - no persistence
py:call(math, sqrt, [16]).
```
Process-bound environments allow both stateless and stateful patterns in the same system.
## Technical Details
Environments are stored as NIF resources with the following lifecycle:
1. **Creation**: First `py:exec/eval/call` in a process allocates an environment
2. **Storage**: Reference kept in process dictionary under `py_local_env`
3. **Usage**: Each call uses the environment as local namespace
4. **Cleanup**: NIF resource destructor runs when process terminates
For subinterpreters, environments are created inside the target interpreter using its memory allocator - critical for memory safety.
### Interpreter ID Validation
Each `py_env_resource_t` stores the Python interpreter ID (`interp_id`) when created. For OWN_GIL contexts, before any operation using a process-local env, the system validates that the env belongs to the current interpreter:
```c
PyInterpreterState *current_interp = PyInterpreterState_Get();
if (penv->interp_id != PyInterpreterState_GetID(current_interp)) {
return {error, env_wrong_interpreter};
}
```
This prevents:
- Using an env from a destroyed interpreter (dangling pointer)
- Using an env created for a different OWN_GIL context
- Memory corruption from cross-interpreter dict access
### Cleanup Safety
For the main interpreter (`interp_id == 0`), the destructor acquires the GIL and decrefs the Python dicts normally.
For subinterpreters, the destructor skips `Py_DECREF` because:
1. `PyGILState_Ensure` cannot safely acquire a subinterpreter's GIL
2. The Python objects will be freed when the subinterpreter is destroyed via `Py_EndInterpreter`
This design prioritizes safety over avoiding minor memory leaks during edge cases.
## See Also
- [OWN_GIL Internals](owngil_internals.md) - Architecture and safety mechanisms for OWN_GIL mode
- [Scalability](scalability.md) - Mode comparison (owngil vs subinterp vs worker)
- [Event Loop Architecture](event_loop_architecture.md) - Per-process namespace management
- [Context Affinity](context-affinity.md) - Context binding and routing
- [Scheduling](asyncio.md) - Cooperative scheduling for long operations
