Skip to main content

docs/blog-dual-gpu-demo.md

# Two GPUs, Two FreeBSD Boxes, One Erlang Cluster

*A dual-GPU Bayesian inference demo that nobody asked for, built
from parts that cost less than lunch.*

---

## The setup

Two 2013 Mac Pros sitting on a shelf. Both running FreeBSD 15.0.
Both with decade-old NVIDIA Kepler GPUs. Connected by a $5
ethernet cable to the same LAN switch.

| Node | Host | GPU | VRAM |
|------|------|-----|------|
| gpu1@192.168.0.248 | mac-248 | GT 750M | 2GB |
| gpu2@192.168.0.247 | mac-247 | GT 650M | 1GB |

Total GPU investment: $0 (surplus hardware).

## The demo

From mac-248's terminal:

```elixir
Node.connect(:"gpu2@192.168.0.247")

# Build IR + sample on BOTH GPUs in parallel
# gpu1: local Vulkan dispatch
# gpu2: remote via :rpc.call → Vulkan dispatch on 247's GPU
```

Output:

```
=== DUAL-GPU MCMC DEMO ===
gpu1: NVIDIA GeForce GT 750M @ mac-248
gpu2: NVIDIA GeForce GT 650M @ mac-247
GT 750M: 934ms, 200 samples
GT 650M: 1278ms, 200 samples
Combined 400 samples: mean=-0.081
=== TWO GPUs x TWO FreeBSD x ONE ERLANG CLUSTER ===
```

400 NUTS samples of Normal(0,1). Two independent chains on two
GPUs on two machines. Combined posterior mean: -0.081 (expected
~0.0). 1.3 seconds wall time.

## How it works

### The transport: Erlang distribution

No gRPC. No REST. No message queues. No protobuf. No Kubernetes.

```elixir
Node.connect(:"gpu2@192.168.0.247")
:rpc.call(:"gpu2@192.168.0.247", Exmc.NUTS.Sampler, :sample, [ir, %{}, opts])
```

Two lines. The Erlang VM handles TCP connection, serialization,
authentication (via a shared cookie), and result return. The cookie
is the entire security model:

```sh
# mac-248:
elixir --name gpu1@192.168.0.248 --cookie zed_gpu_demo -S mix

# mac-247:
elixir --name gpu2@192.168.0.247 --cookie zed_gpu_demo -S mix
```

Same cookie = same cluster. Different cookie = invisible.

### The compute: Vulkan fused chain shaders

Each GPU runs a fused leapfrog chain shader — one SPIR-V compute
shader that performs K=32 consecutive NUTS leapfrog steps in a
single GPU dispatch. Closed-form Normal gradient baked into the
shader. No autodiff, no graph compilation, no JIT warmup.

The shader was written in GLSL, compiled to SPIR-V by
`glslangValidator` on FreeBSD, vendored as a `.spv` binary.
Identical on both machines. The Vulkan driver loads it at
runtime.

### The key insight: build IR on the remote node

Nx tensors backed by Vulkan GPU memory can't be serialized
across Erlang distribution — the GPU buffer ref is a local
pointer. So the remote dispatch builds the model IR **on the
remote node**:

```elixir
:rpc.call(:"gpu2@192.168.0.247", :erlang, :apply, [fn ->
  ir = Exmc.Builder.new_ir()
       |> Exmc.Builder.rv("x", Exmc.Dist.Normal,
            %{mu: Nx.tensor(0.0), sigma: Nx.tensor(1.0)})
  {trace, _} = Exmc.NUTS.Sampler.sample(ir, %{}, opts)
  [{_, samples}] = Enum.to_list(trace)
  Nx.to_flat_list(samples)  # return plain floats, not GPU tensors
end, []])
```

The closure is serialized (just code, no GPU state). The sampling
runs entirely on gpu2's BEAM + GPU. The result — a list of
floats — is sent back over Erlang distribution. The GPU memory
stays on the remote node.

## What we had to fix

### OTP version alignment

mac-248 ran Erlang/OTP 27 + Elixir 1.18.4. mac-247 ran OTP 26 +
Elixir 1.17.3. The Rustler NIF compiled against OTP 26 didn't
export all functions — specifically `upload_binary_into/2` which
the persistent-buffer optimization needs.

Fix: installed OTP 27 runtime (`pkg install erlang-runtime27`)
and built Elixir 1.18.4 from source (`gmake && gmake install`)
on mac-247. Both machines now run the same BEAM.

### Rustler path dep NIF caching

When Zed (our deployment tool) uses nx_vulkan as a path dep,
Rustler 0.36's Cargo target cache doesn't always pick up new
NIF functions. The compiled `.so` has the symbols but the BEAM
module's `on_load` callback rejects it.

Workaround: compile the NIF once in the source project
(`cd ~/nx_vulkan && mix compile`), then copy the `.so` into the
consumer project's `_build`. Or run the demo directly from the
exmc project directory, bypassing the path dep.

### pf firewall setup

BEAM distribution uses EPMD (port 4369) plus a dynamic port for
actual node communication. We pinned the distribution port range:

```sh
--erl "-kernel inet_dist_listen_min 9100 inet_dist_listen_max 9200"
```

And opened those ports in pf on mac-247:

```
pass in proto tcp from 192.168.0.0/24 to any port 4369
pass in proto tcp from 192.168.0.0/24 to any port 9100:9200
```

### Named nodes required

`mix run` starts an anonymous (unnamed) BEAM node. Anonymous
nodes can't participate in Erlang distribution — `Node.connect`
silently fails. The fix: always use `--name node@ip`.

## The numbers

### Per-GPU performance

| GPU | 200+200 NUTS | ms/iter |
|-----|-------------|---------|
| GT 750M (mac-248) | 934 ms | 2.3 ms |
| GT 650M (mac-247) | 1,278 ms | 3.2 ms |

The GT 650M is ~37% slower — consistent with fewer CUDA cores
(384 vs 384, but lower clock) and Gen2 vs Gen2 PCIe.

### Per-fence Vulkan driver latency

| Metric | FreeBSD GT 750M |
|--------|-----------------|
| vkQueueSubmit | 11.6 µs |
| vkWaitForFences | 406 µs |
| Command record | 4.3 µs |
| **Per-dispatch** | **422 µs** |

For comparison, Linux NVIDIA on an RTX 3060 Ti: 1,287 µs per
dispatch. FreeBSD's driver is 3.1× faster on fence waits.

### Fused vs unfused speedup

| Path | ms/iter | Speedup |
|------|---------|---------|
| Unfused (per-op dispatch) | 283 ms ||
| Fused chain (K=32) | 3.3 ms | **86.7×** |

The chain shader reduces ~384 fence waits to ~4 per iteration.

## What this proves

1. **BEAM distribution is GPU dispatch fabric.** `:rpc.call` +
   Erlang cookies is the entire transport layer for multi-GPU
   MCMC. No custom serialization protocol. No service mesh.

2. **FreeBSD is a real GPU compute platform.** Not "it boots" —
   it runs 2000 NUTS iterations in 1 second on decade-old
   hardware, with a 3.1× driver advantage over Linux.

3. **Surplus hardware is compute hardware.** Two Mac Pros from
   2013, destined for recycling, now run a distributed Bayesian
   inference cluster. The GPUs have 384 CUDA cores each. They
   work.

4. **The architecture composes.** Same chain shader on both GPUs.
   Same eXMC sampler. Same Erlang distribution protocol. Add a
   third Mac Pro with a GPU → three-node cluster, no code changes.

## The stack

```
Elixir (eXMC NUTS sampler)
  ↓ :rpc.call (Erlang distribution, TCP, cookies)
Nx.Vulkan.Backend (Elixir, Rustler NIF)
  ↓ extern "C" FFI
spirit Backend_par_vulkan (C++)
  ↓ Vulkan API
NVIDIA driver (FreeBSD nvidia-driver-470)
  ↓ PCIe
GPU (Kepler, 2013)
```

Seven layers. Same SPIR-V binary on both machines. Same Elixir
code on both machines. The only difference: the IP address in
`Node.connect`.

## Reproduction

On any two FreeBSD machines with NVIDIA GPUs:

```sh
# Machine A:
pkg install vulkan-loader erlang rust
cd ~/nx_vulkan && mix compile && mix test  # 152/0
cd ~/exmc/exmc && mix compile
elixir --name gpu1@<A_IP> --cookie demo \
  --erl "-kernel inet_dist_listen_min 9100 inet_dist_listen_max 9200" \
  -S mix run --no-halt

# Machine B (same steps, different name):
elixir --name gpu2@<B_IP> --cookie demo \
  --erl "-kernel inet_dist_listen_min 9100 inet_dist_listen_max 9200" \
  -S mix run --no-halt

# From A's iex:
Node.connect(:"gpu2@<B_IP>")
# ... dispatch sampling via :rpc.call ...
```

Three commands per machine. No Docker. No Ansible. No YAML.

---

*Two GPUs. Two FreeBSD boxes. One Erlang cluster. 400 NUTS samples
in 1.3 seconds. mean=-0.081.*

*Built with parts that cost less than lunch, on an OS that nobody
uses, with a GPU API that nobody chose, connected by a protocol
from 1986.*

*It works.*