# `vfs` — protocol-based virtual filesystem for the Elixir AI tools stack
A new library that replaces `feat/mountable-virtual-filesystem-core` in `just_bash` and becomes a shared dep with `pyex`, so an agent can hand the same filesystem state back and forth between bash execution and Python interpretation.
> **Name**: `VFS` — module acronym stays uppercased everywhere, never `Vfs` (matches the `JustBash.FS` convention from `feat/git-fs` commit `6250b8f` "uppercase FS acronym in new module names"). Hex package: `:vfs` — available on hex.pm.
>
> **Repo**: `ivarvong/vfs`.
>
> **Status**: design decisions settled below; staff-reviewed by @daveLucia.
>
> **Amended 2026-06-10** (pre-0.1.0 release): code sketches below were written before implementation and have been updated to match the shipped surface — the 10-callback protocol, `%VFS.Error{}` struct returns (the draft used bare atoms), the trimmed capability set, and no `grep`/`glob`/`cp`/`mv` helpers in the library (consumer-side compositions). The decision table and rationale are original. Where this document and the code disagree, that is a bug — see CLAUDE.md.
>
> | Decision | Choice |
> |---|---|
> | Protocol name | `VFS.Mountable` (renamed from `VFS.Filesystem` per Dave — "filesystem" is overloaded) |
> | Dispatch substrate | Single protocol, not behaviours |
> | Reads thread state back | Yes (every op returns updated impl) |
> | Mount-table-as-backend | `%VFS{}` itself implements the protocol |
> | Primary read API | Lazy: `stream_read/3` (Enumerable). `read_file/2` is a derived helper on `VFS`, not a protocol callback. |
> | Streaming/pushdown | First-class in v1 (`stream_read`, `walk`, `materialize`) |
> | Stat shape | `%VFS.Stat{type, size, mtime, mode}` — own struct, shaped to virtual-FS semantics. Not `File.Stat` (OS-fs concepts like `inode`/`uid`/`gid` don't apply to git blobs or S3 objects). Follows stdlib `type: atom()` convention. |
> | Errors | Structured `%VFS.Error{kind, path, mount, message}` exception. Pattern match on `:kind` for control flow. Kind atoms follow POSIX (`:enoent`, `:eisdir`, ...). |
> | Read-side primitives | `walk/3` + `stream_read/3`. Sufficient for any bulk operation: grep, mapreduce, fulltext, sync, dedup, backup. |
> | Higher-order ops | Out of v1 core. `grep`/`glob`/`cp`/`mv` belong in a companion package or consumer code; the library stays small on purpose. |
> | Cross-backend pushdown | Deferred to optional secondary protocols (e.g. `VFS.Searchable`, `VFS.ContentAddressed`) when a pattern recurs. v1 has none. |
> | Read/write protocol | Single combined protocol; not split. Read-only-ness is a `capabilities/1` property. |
> | Runtime deps | `:telemetry` only (effectively-stdlib in Elixir; required for the agent-loop observability story). |
> | Cut from v0.1 protocol | `lstat`, `readlink`, `symlink`, `link`, `chmod`, `append_file` — no v1 backend uses them; YAGNI. Add when there's a real consumer. |
> | `VFS.Git` | **Not in this library.** Lives as `defimpl VFS.Mountable, for: Exgit.Repository` inside `:exgit`. |
> | `VFS.Overlay` / `VFS.ReadOnly` | **Not stock impls.** Documented as worked examples; users compose when needed. |
> | S3 backend | Deferred from v1 |
> | Streaming writes | Deferred to v0.2 |
> | Path watching | Out of scope; separate protocol if/when needed |
---
## TL;DR
1. **One protocol, `VFS.Mountable`.** Every backend is a struct that `defimpl`s it. No behaviours. The current code in both `just_bash` and `pyex` already does fake protocol dispatch via `fs.__struct__.read(fs, ...)` — switching to actual protocols deletes that fake dispatch and gives consolidated, fast dispatch tables.
2. **Reads thread state back, not just writes.** This is the single biggest defect in the existing `JustBash.FS.Backend` behaviour and the reason `GitFS.materialize/1` had to exist as a workaround. With `read_file(impl, path) :: {:ok, binary, impl} | {:error, reason}`, a lazy `GitFS` cache is preserved across reads.
3. **Mount-table-as-FS.** The `VFS.t()` struct (mount table + dispatch) itself implements `VFS.Mountable`. Mount tables nest. `LayeredFS` doesn't need to know whether its inner is a single backend or a mount table.
4. **Lazy by default.** `stream_read/3` is the protocol's primary read primitive (returns `Enumerable.t()`). `read_file/2` is a thin helper that runs the stream into a binary. Backends with bytes already in hand return a single-chunk stream from `stream_read`; every backend gets `read_file` for free. This is the substantive shift from the previous draft, where `read_file` was primary and `stream_read` was bolted on.
5. **Tiny v1 surface.** Library ships: protocol, `%VFS{}` mount table, `VFS.Memory`, `VFS.Path`, `VFS.Stat`, `VFS.Error`, `VFS.StreamOptions`, `VFS.Skeleton`, `VFS.Default`. **No `VFS.Git`** (lives as a `defimpl` inside `:exgit`). **No `VFS.Overlay` / `VFS.ReadOnly`** (documented patterns, not stock impls). Caller-provided is one `defimpl`.
6. **Two primitives, anything composes on top.** `walk/3` (lazy tree traversal yielding `{path, stat}`) and `stream_read/3` (lazy per-file byte stream) are the universal read-side primitives. Together they're sufficient to build grep, mapreduce, fulltext indexing, content-addressed dedup, sync, backup, stats — any bulk operation a consumer wants. `grep` and `glob` are consumer-side compositions (or a future companion package), not protocol ops and not library helpers; `%VFS.Match{}` was a smell because it was an op-specific result type leaking into the universal interface. Backend-specific perf optimizations live as backend-specific functions (e.g. `Exgit.FS.grep/4`); cross-backend pushdown for a recurring pattern gets a future optional secondary protocol (`VFS.Searchable`, `VFS.ContentAddressed`, etc.) — not pollution of the core.
7. **Both consumer libs converge on this protocol.** `just_bash` deletes its `FS.Backend` behaviour and its in-memory/RO impls; `pyex` deletes `Pyex.Filesystem` and its `Memory` impl. Both now hold a `VFS.t()` and pass it back to the caller after every operation.
---
## Why protocols, not behaviours (evidence)
The receipts are in the existing code:
```elixir
# pyex/lib/pyex/ctx.ex:835
mod = fs.__struct__
case mod.read(fs, path) do ...
# pyex/lib/pyex/ctx.ex:905
case fs.__struct__.write(fs, path, buffer, mode) do ...
# just_bash/lib/just_bash/fs/fs.ex:82
def mount(%__MODULE__{mounts: mounts} = fs, mountpoint, backend_state) do
module = backend_state.__struct__
...
end
```
This is "I want polymorphism on a struct value" — which is exactly what protocols are for. The behaviour requires every caller to either pre-bundle `{module, state}` tuples or extract the module from `__struct__` at the call site. Both repos do the latter, repeatedly. With a protocol:
```elixir
VFS.Mountable.read_file(fs, path) # dispatched on fs's struct, no __struct__ pokery
```
Other concrete wins for this codebase:
- **Caller-provided backends become genuinely cheap.** Drop a `defimpl` in your app, done. With behaviours, you're additionally responsible for wiring up `__struct__` extraction or `{mod, state}` plumbing.
- **Protocol consolidation in releases** turns dispatch into compile-time-known function calls — relevant for the hot path in an agent loop (every tool call hits the FS).
- **Decorators don't need to know the inner type.** A user-written read-only or CoW overlay holds an `inner :: term()` (any protocol-implementing struct) and delegates via the protocol. The current `JustBash.FS.ReadOnlyFS` has to store `inner_mod` separately precisely because behaviours don't dispatch on values. (Decorators are documented patterns, not stock impls — see "Worked example patterns" below — but the protocol-vs-behaviour delta is what makes them one-field structs instead of two.)
The one thing protocols give up vs. behaviours: dialyzer can't enforce that an impl exists for every type at compile time. Mitigation: `mix.exs` has `consolidate_protocols: Mix.env() != :test`, and we ship a `VFS.assert_implemented!/1` that raises with a helpful message if a struct shows up without an impl. Plus the impls are all internal struct types we control, so this is mostly a non-issue in practice.
---
## Library shape
```
vfs/
├── lib/
│ ├── vfs.ex # Mount-table struct + public API + `defimpl VFS.Mountable, for: VFS`
│ ├── vfs/
│ │ ├── mountable.ex # The protocol
│ │ ├── path.ex # Pure path utilities (normalize, dirname, basename, resolve)
│ │ ├── stat.ex # %VFS.Stat{type, size, mtime, mode}
│ │ ├── memory.ex # In-memory backend (the only stock impl)
│ │ ├── default.ex # Default `walk` impl used by Skeleton
│ │ ├── error.ex # %VFS.Error{kind, path, mount, message} exception
│ │ ├── stream_options.ex # chunk_size / byte_range / line_range handling
│ │ └── skeleton.ex # `use VFS.Skeleton` macro for impl authors
└── mix.exs
# mix.exs deps — none required at runtime
# (consumers add :exgit themselves; exgit ships defimpl VFS.Mountable, for: Exgit.Repository)
```
The library has zero non-stdlib runtime deps. `:exgit` takes `:vfs` as an optional dep and ships its own `defimpl`; same pattern for any future S3, FTP, IPFS, etc. backend. This inverts the dependency direction from the previous draft, where vfs knew about exgit; now exgit knows about vfs (which is correct — vfs is the abstraction, exgit is the concrete thing). Per Dave: "I could see the behavior for VFS implemented by ExGit instead. Protocols make this possible."
---
## The protocol
### Result type — `%VFS.Stat{}`
```elixir
defmodule VFS.Stat do
@moduledoc """
Metadata for a path in a virtual filesystem.
Deliberately *not* `File.Stat` from stdlib: that struct is shaped around
POSIX `stat(2)` for real OS files (`inode`, `uid`, `gid`, `links`,
`major_device`, `minor_device`). For a virtual filesystem where most
backends are git blobs, S3 objects, or in-memory maps, those fields are
meaningless and would be `nil` constantly. Better to have a struct
shaped to the abstraction.
Field conventions follow `File.Stat` where they exist (`type: atom()`,
not `is_file: boolean()`).
"""
@type t :: %__MODULE__{
type: :regular | :directory | :symlink | :other,
size: non_neg_integer(),
mtime: DateTime.t(),
mode: non_neg_integer() | nil
}
defstruct [:type, :size, :mtime, mode: nil]
end
```
Four fields, no more:
- **`type`** — atom, matching `File.Stat`'s convention. Use `stat.type == :regular`, never `is_file`.
- **`size`** — bytes for files, undefined-but-present for directories (backends usually return 0).
- **`mtime`** — `DateTime.t()`, not Erlang datetime tuple. VFS is a new abstraction; no legacy compatibility burden, so we canonicalize on the modern type. Backends without real mtimes (e.g. content-addressed git blobs) use a deterministic value such as the commit time of the containing tree, or epoch.
- **`mode`** — POSIX permission bits when meaningful (Memory tracks them, exgit returns the tree-entry mode), `nil` when not (S3, in general). Optional and explicitly nullable.
Things deliberately omitted, with reasoning:
- **`atime`, `ctime`** — neither git nor S3 nor an in-memory store has a meaningful access or change time. Real OS-file callers can read `File.stat!/1` directly.
- **`inode`, `links`, `uid`, `gid`, `major_device`, `minor_device`** — POSIX-fs concepts that don't generalize. A git blob has no inode. An S3 object has no uid.
- **`is_symbolic_link: boolean()`** — `type == :symlink` covers it.
If a future backend needs to expose extra metadata (e.g. an exgit mount wanting to expose blob SHA), that's a backend-specific concern; the backend module exposes its own getter (`Exgit.FS.blob_sha(repo, ref, path)`). We don't grow the universal struct for one backend's needs — same principle as keeping `grep` out of the protocol.
### Protocol
```elixir
defprotocol VFS.Mountable do
@moduledoc """
Pluggable virtual filesystem. Implementations are plain structs; the
protocol dispatches on the struct type.
## Path contract
All paths are absolute, already normalized, leading `/`. Backends behave
as if rooted at `/`. Mount-prefix stripping happens in `VFS` before the
call reaches a backend impl.
## State threading
*Every* operation — including reads — returns the (possibly updated)
impl as the last element of the success tuple. Lazy backends (e.g. an
exgit-backed mount with a partial-clone repo) cache fetched blobs in
their struct on read; throwing the updated struct away (as the previous
behaviour-based design did) destroyed those caches. Callers thread the
new state forward.
## Errors
Structured `%VFS.Error{kind, path, mount, message}` exceptions. `:kind`
follows POSIX: `:enoent`, `:eexist`, `:eisdir`, `:enotdir`, `:erofs`,
`:eacces`, `:einval`, `:exdev`, `:eio`, `:eloop`, plus `:enotsup` for
backends that don't support an op.
"""
@type t :: struct()
@type path :: String.t()
# ── queries — return state because lazy backends mutate cache on read ──
@spec exists?(t, path) :: {boolean, t}
def exists?(impl, path)
@spec stat(t, path) :: {:ok, VFS.Stat.t(), t} | {:error, VFS.Error.t()}
def stat(impl, path)
# Bounded backends return a list of names; paginated/unbounded backends
# return a Stream. Consumers treat the result as an Enumerable.
@spec readdir(t, path) :: {:ok, Enumerable.t(String.t()), t} | {:error, VFS.Error.t()}
def readdir(impl, path)
# ── streaming reads are the primary read API ──
#
# `stream_read/3` is the protocol's only file-content read primitive.
# `VFS.read_file/2` (the helper, not a protocol op) runs this stream into
# a binary for callers who want eager bytes; backends with a natural eager
# path return a single-chunk stream. Per Dave: file reads return
# "something you could pull out lazily as well" — making the lazy form
# primary means callers never pay for a 1 GiB blob materialization they
# didn't ask for.
#
# The Enumerable emits binary chunks. opts: :chunk_size (default 64 KiB),
# :byte_range, :line_range. `t` returned in the success tuple is the
# impl after any header/metadata reads needed to *open* the stream;
# cache state populated *during* enumeration does not escape the stream
# (see "cache-eviction caveat" below).
@spec stream_read(t, path, keyword) :: {:ok, Enumerable.t(binary), t} | {:error, VFS.Error.t()}
def stream_read(impl, path, opts)
# ── streaming tree walk ──
# Emits {path, %VFS.Stat{}}. opts: :max_depth, :include_dirs (default
# false). Returns a bare Enumerable, not a {:ok, _, t} tuple — the one
# exception to state threading. `glob` and `grep` are NOT protocol ops —
# they're consumer-side compositions of this + `stream_read`. Keeping the
# protocol minimal avoids forcing higher-order result shapes (e.g. a
# grep-specific `%VFS.Match{}`) into the universal interface.
@spec walk(t, path, keyword) :: Enumerable.t({path, VFS.Stat.t()})
def walk(impl, root, opts)
# ── eager prefetch lever for lazy backends ──
# No-op for Memory; e.g. Exgit.Repository.materialize/2 for an Exgit-backed
# mount. Useful when callers know they're about to do a full-tree scan and
# want to pay the network cost up front rather than per-blob during
# enumeration.
@spec materialize(t, keyword) :: {:ok, t} | {:error, VFS.Error.t()}
def materialize(impl, opts)
# ── mutations ──
@spec write_file(t, path, binary, keyword) :: {:ok, t} | {:error, VFS.Error.t()}
def write_file(impl, path, content, opts)
@spec mkdir(t, path, keyword) :: {:ok, t} | {:error, VFS.Error.t()}
def mkdir(impl, path, opts)
@spec rm(t, path, keyword) :: {:ok, t} | {:error, VFS.Error.t()}
def rm(impl, path, opts)
# ── capability introspection — lets callers fast-path or refuse ──
@spec capabilities(t) :: MapSet.t(capability)
def capabilities(impl)
end
```
Ten callbacks — that is the entire shipped surface. The draft additionally
sketched `lstat`, `readlink`, `read_file`-as-callback, `append_file`,
`chmod`, `symlink`, and `link`; all were cut before 0.1 (no v1 backend used
them — see the decision table).
Backends that don't support an op return `{:error, %VFS.Error{kind: :enotsup}}`. `capabilities/1` reports the set so callers can avoid trying. Capability atoms: `:read`, `:write`, `:mkdir` (write does not imply mkdir — flat-keyed backends like S3 support `:write` without it), plus pushdown/streaming markers `:native_walk`, `:native_stream_read`, and `:lazy` (the impl benefits from `materialize/2` before bulk reads).
### Skeleton macro for impl authors
```elixir
defmodule VFS.Skeleton do
@moduledoc """
Default impls of the optional `VFS.Mountable` ops a backend doesn't
override. `use VFS.Skeleton` inside a `defimpl` block.
The required minimum for any backend: `stream_read/3`, `readdir/2`,
`stat/2`, `exists?/2`, `write_file/4`, `mkdir/3`, `rm/3`,
`capabilities/1` (read-only backends refuse the mutations with :erofs).
The skeleton supplies `walk/3` and `materialize/2`.
"""
defmacro __using__(_opts) do
quote do
# ── walk default composed from readdir + stat ──
def walk(impl, root, opts), do: VFS.Default.walk(impl, root, opts)
def materialize(impl, _opts), do: {:ok, impl}
defoverridable walk: 3, materialize: 2
end
end
end
```
The default `walk` lives in `VFS.Default.walk/3` — a lazy depth-first traversal that recursively `readdir`s. The eager read is *not* a protocol op or skeleton default: `VFS.read_file/2` (the public helper) runs `stream_read/3` into a binary, so every backend gets it for free and backends with bytes already in hand return a single-chunk stream. The cache-eviction caveat (cache state populated *during* enumeration doesn't escape the stream) is documented below; `materialize/2` is the lever for callers who need it pre-populated.
### Example: how `:exgit` ships a defimpl (lives in exgit, not vfs)
Exgit takes `:vfs` as an optional dep and ships `defimpl VFS.Mountable, for: Exgit.Repository`. Per Dave: *"I could see the behavior for VFS implemented by ExGit instead. Protocols make this possible."* This is exactly the case protocols are for — vfs declares the abstraction, exgit attaches an impl directly to its own `Repository.t()` struct, no wrapper, no shim.
```elixir
# THIS LIVES IN :exgit, NOT in :vfs (illustrative sketch)
defimpl VFS.Mountable, for: Exgit.Repository do
use VFS.Skeleton
# writes refused — git is read-only via this protocol
def write_file(repo, path, _, _), do: {:error, VFS.Error.new(:erofs, path: path)}
def mkdir(repo, path, _), do: {:error, VFS.Error.new(:erofs, path: path)}
def rm(repo, path, _), do: {:error, VFS.Error.new(:erofs, path: path)}
# streaming read — the primary read primitive
def stream_read(%Exgit.Repository{} = repo, path, opts) do
ref = repo.default_ref
case Exgit.FS.read_path(repo, ref, path) do
{:ok, {_mode, %Exgit.Object.Blob{data: data}}, repo2} ->
chunk_size = Keyword.get(opts, :chunk_size, 64 * 1024)
stream = data |> chunk_binary(chunk_size)
{:ok, stream, repo2}
{:error, reason} -> {:error, VFS.Error.new(map_error(reason), path: path)}
end
end
# streaming pushdown — walk traverses tree objects without inflating blobs
def walk(%Exgit.Repository{} = repo, _root, _opts),
do: Exgit.FS.walk(repo, repo.default_ref)
# exgit's walk requires an eager repo when scanning the full tree —
# materialize/2 is the lever
def materialize(%Exgit.Repository{} = repo, _opts) do
case Exgit.Repository.materialize(repo, repo.default_ref) do
{:ok, repo2} -> {:ok, repo2}
err -> err
end
end
# ...stat, exists?, readdir wrap Exgit.FS too...
def capabilities(_), do: MapSet.new([:read, :native_walk, :lazy])
end
```
Two things to note about what the defimpl does *not* contain:
1. **No `grep`/`glob` impls.** Those aren't protocol ops (or library helpers). Consumers compose `walk + stream_read + line scan` — correct, lazy, memory-bounded; see "Worked example 1" below.
2. **The pack-internal grep optimization remains accessible as `Exgit.FS.grep/4`** — a backend-specific function on the exgit side. Power users who have an `Exgit.Repository` in hand and need maximum performance call it directly. This is the "escape hatch" pattern: the protocol gives you correctness and abstraction; the backend module gives you peak performance when you need it.
The 1M-file `grep -r TODO /repo` agent case via the protocol: a consumer grep walks tree objects (no blob fetches), then for each path calls `stream_read` (one blob at a time, line-scanned, discarded). Memory bounded to one blob. Correct. Slower than `Exgit.FS.grep/4`'s pack scanner, but the latter is a perf optimization the abstraction doesn't need to absorb.
---
## Primitives — `walk` + `stream_read`, and what builds on them
The protocol's read-side surface is built around two primitives:
- **`walk/3`** — lazy tree traversal yielding `Stream.t({path, %VFS.Stat{}})`. Backend-specific implementations control how cheaply this can be done (exgit walks tree objects without inflating blobs; Memory walks an in-memory map; a future S3 backend uses paginated `ListObjectsV2`).
- **`stream_read/3`** — lazy per-file byte stream yielding chunks. Includes `:byte_range` and `:line_range` opts for partial reads.
Together these are sufficient for any bulk read-side operation a consumer wants to build. The motivating scenario was "1M-file grep on an exgit-backed mount," but the primitives weren't designed for grep — they're designed for *any* bulk traversal that needs to stay memory-bounded and lazy on the per-file axis. Below: three worked examples showing the same primitives compose into different consumer-side operations.
### Worked example 1: `grep` (consumer-side, not in the library)
```elixir
def grep(fs, root, pattern, opts \\ []) do
fs
|> VFS.Mountable.walk(root, opts) # Stream.t({path, stat})
|> Stream.filter(fn {_, stat} -> stat.type == :regular end)
|> Stream.flat_map(fn {path, _stat} ->
case VFS.Mountable.stream_read(fs, path, []) do
{:ok, byte_stream, _fs2} -> scan_lines(byte_stream, pattern, path, opts)
_ -> []
end
end)
end
```
Returns `Stream.t({path, line_number, line, before_context, after_context})`. Plain tuples — no protocol-level result struct. Memory-bounded: at any moment, at most one file's content is being held.
### Worked example 2: a mapreduce framework
The exact thing Ivar asked about. A consumer can write this as a library on top of `:vfs` without the protocol changing:
```elixir
defmodule MyApp.MapReduce do
@moduledoc """
Parallel map-reduce over a VFS. Worker fan-out via Task.async_stream;
no special protocol support needed beyond walk + stream_read.
"""
def run(fs, root, map_fn, reduce_fn, acc, opts \\ []) do
concurrency = Keyword.get(opts, :max_concurrency, System.schedulers_online() * 2)
fs
|> VFS.Mountable.walk(root, opts)
|> Stream.filter(fn {_, stat} -> stat.type == :regular end)
|> Task.async_stream(
fn {path, stat} ->
{:ok, content_stream, _} = VFS.Mountable.stream_read(fs, path, [])
map_fn.(path, stat, content_stream)
end,
max_concurrency: concurrency,
ordered: false
)
|> Stream.flat_map(fn
{:ok, result} -> [result]
{:exit, _reason} -> []
end)
|> Enum.reduce(acc, reduce_fn)
end
end
# Usage: count word frequencies across an entire repo
MyApp.MapReduce.run(
fs,
"/repo",
fn _path, _stat, content_stream ->
content_stream
|> Stream.flat_map(&String.split(&1, ~r/\W+/))
|> Enum.frequencies()
end,
&Map.merge(&1, &2, fn _k, a, b -> a + b end),
%{}
)
```
This works on `VFS.Memory`, on `Exgit.Repository`-backed mounts, on a CoW overlay over either, on the `%VFS{}` mount table — anywhere the protocol is implemented. The consumer doesn't care about backend identity. **`walk` and `stream_read` carry the entire weight.**
### Worked example 3: stats / dedup-by-content-hash
```elixir
# Total bytes by extension
fs
|> VFS.Mountable.walk("/", [])
|> Stream.filter(fn {_, s} -> s.type == :regular end)
|> Enum.reduce(%{}, fn {path, stat}, acc ->
ext = Path.extname(path)
Map.update(acc, ext, stat.size, &(&1 + stat.size))
end)
# Group files by content hash (dedup)
fs
|> VFS.Mountable.walk("/", [])
|> Stream.filter(fn {_, s} -> s.type == :regular end)
|> Task.async_stream(fn {path, _} ->
{:ok, stream, _} = VFS.Mountable.stream_read(fs, path, [])
hash = stream |> Enum.reduce(:crypto.hash_init(:sha256), &:crypto.hash_update(&2, &1)) |> :crypto.hash_final()
{hash, path}
end)
|> Stream.map(fn {:ok, x} -> x end)
|> Enum.group_by(&elem(&1, 0), &elem(&1, 1))
|> Enum.filter(fn {_, paths} -> length(paths) > 1 end)
```
Both compositions, no protocol changes.
### When to add a new primitive
The bar for adding a new protocol op: **it can't be expressed efficiently in terms of `walk + stream_read + stat`, and multiple backend types could meaningfully implement it differently.** Some candidates that have come up and where they currently sit:
| Operation | Status | Reasoning |
|---|---|---|
| `glob` (declarative path filter) | Consumer-side, not protocol op | Composes from `walk + path match`. Could become a `:glob` *option* on `walk` in v0.2 if backends with metadata indexes (sqlite-backed, S3) want pushdown. |
| `hash(path)` (content-addressed) | Not in v1 | Git has it natively; nobody else does yet. Future `VFS.ContentAddressed` optional protocol when a second backend gains native hashes. |
| `diff(fs1, fs2)` | Not in v1 | Cross-FS, expensive in general. Caller composes from two walks + comparison. Future `VFS.Diffable` if it recurs. |
| `find(predicate)` (stat-based) | Not in v1 | Caller `Stream.filter`s walk output. Pushdown only matters for backends with metadata indexes; deferred until that comes up. |
The principle: **start with the smallest sufficient set of primitives. Add ops only when the composed version is provably wrong (incorrect, leaks, blows memory) for some real backend.**
### The pushdown escape hatch
For maximum performance on a specific backend, users call backend functions directly:
```elixir
# Goes through the abstraction (correct, slower but bounded memory):
MyApp.Grep.run(fs, "/repo", "TODO") |> Enum.to_list() # walk + stream_read composition
# Bypasses the abstraction for max perf (only works on exgit-backed):
Exgit.FS.grep(repo, ref, "TODO", []) |> Enum.to_list()
```
Backend-specific functions are not "leaks" in the abstraction — they're the right place for performance optimizations that don't generalize. The protocol gives portability and correctness; the backend module gives peak perf when the user can commit to a specific backend.
### The materialize lever (for lazy backends)
A lazy partial-clone exgit repo will, during a full-tree `walk + stream_read`, fetch blobs on demand — one round trip per blob. Callers who know they're about to iterate the whole tree can pre-warm:
```elixir
{:ok, fs} = VFS.Mountable.materialize(fs, []) # cheap for non-lazy mounts; pulls blobs for exgit-backed ones
results = MyApp.MapReduce.run(fs, "/", map_fn, reduce_fn, acc)
```
`VFS.Memory`'s `materialize/2` impl is a no-op. An overlay's `materialize/2` recurses into both layers. The mount-table `materialize/2` fans out. Uniform call; only lazy backends actually do network work.
### The cache-eviction caveat
State threading and lazy enumerables don't compose cleanly: a `Stream` from `walk` captures `impl` in a closure, but cache updates that happen inside the stream (e.g. an exgit-backed mount's blob cache populating during `stream_read` calls) don't escape back to the caller. Two ways to handle this:
1. **For one-shot ops** (CLI-style `grep`, mapreduce-then-discard), accept the eviction. The result is still correct — the next operation just won't see the cache the stream populated.
2. **For agent loops** that iterate then re-touch the same files, call `materialize/2` first. Subsequent `read_file` / `stream_read` calls land in the populated cache.
We deliberately don't try to thread state through streams via tricks (`Stream.transform` accumulators, cache-merge callbacks). The cleaner contract: streams are cache-evicting; `materialize/2` is the lever; document and move on.
### What we deliberately don't do in v1
- **`grep`/`glob` anywhere in the library.** Consumer-side compositions of the primitives (originally drafted as `VFS` helpers; cut entirely before 0.1).
- **`%VFS.Match{}` struct.** Plain tuples in helper return values; consumers wrap to their own types if they want one. Avoids leaking op-specific shapes into the core.
- **`VFS.Searchable` / `VFS.ContentAddressed` / `VFS.Diffable` optional protocols.** Deferred to v0.2+ if patterns recur across backends.
- **Streaming writes.** `stream_write/3` is also v0.2 — `Enumerable.t() -> {:ok, impl}`. Default impl will `Enum.into` a buffered `write_file`. Multipart-upload backends override.
- **`tail`-style follow.** Watching a path for changes is a separate concern; `VFS.Watch` if/when needed.
- **Concurrent walk across mounts.** Mount-table iteration concatenates streams sequentially in v1. Parallelizing is a `Task.async_stream` away when a workload shows it matters. (Note: per-file parallelism within a single walk is already available to consumers — see the mapreduce example.)
---
## Mount table — `VFS.t()` itself implements the protocol
```elixir
defmodule VFS do
defstruct mounts: []
@type mount :: {mountpoint :: String.t(), backend :: struct()}
@type t :: %__MODULE__{mounts: [mount()]}
# ── construction & mount management ──
def new, do: %__MODULE__{}
def mount(%__MODULE__{} = vfs, mountpoint, backend), do: ...
def umount(%__MODULE__{} = vfs, mountpoint), do: ...
def mounts(%__MODULE__{} = vfs), do: ...
# ── telemetry-wrapped helpers that delegate to the protocol ──
# read_file/2 (derived from stream_read), stream_read/3, write_file/4,
# mkdir/3, rm/3, exists?/2, stat/2, readdir/2, walk/3, materialize/2,
# capabilities/1 — these exist for ergonomics (callers say
# `VFS.stream_read(fs, ...)`) and carry the :telemetry instrumentation.
#
# `cp`/`mv` were sketched here in the draft and cut: composable from
# read+write at the consumer, and cross-mount semantics (:exdev) are a
# consumer-policy decision.
end
defimpl VFS.Mountable, for: VFS do
# Longest-prefix mount routing, backend state threaded back through the
# mount tuple — exactly the logic in the existing `JustBash.FS`, but
# adapted so reads also produce updated state. Errors bubbling up get
# :path rewritten into the user's namespace and :mount attached.
def stream_read(%VFS{} = vfs, path, opts) do
case VFS.__resolve__(vfs, VFS.Path.normalize(path)) do
{:ok, mountpoint, sub, backend} ->
case VFS.Mountable.stream_read(backend, sub, opts) do
{:ok, stream, new_backend} ->
{:ok, stream, VFS.__put_mount__(vfs, mountpoint, new_backend)}
{:error, err} ->
{:error, err |> VFS.Error.put_path(path) |> VFS.Error.put_mount(mountpoint)}
end
:no_mount ->
{:error, VFS.Error.new(:enoent, path: path)}
end
end
# ... etc — readdir, stat, walk, write_file, mkdir, rm all follow the same shape
end
```
Because `VFS` itself is a `VFS.Mountable`, you can mount a `VFS` *inside* a `VFS`. Useful for namespacing (one tenant's mount table mounted at `/tenants/acme/`), and falls out of the design for free.
---
## Worked example patterns (not in v1, documented for reference)
Per Dave's review point 4 — Overlay and ReadOnly aren't stock impls in v1; they're patterns users compose. Including them here so the patterns are concrete and so future contributors have a reference. Either could land later as a stock impl if usage shows the same shape getting reinvented across consumers.
### CoW overlay — the agent staging pattern
The agent-staging primitive: reads check `upper` first, then `lower`; writes go to `upper`; deletes mark a path as a whiteout. A concrete `JustBash.Sandbox.Overlay` (or `Pyex.Sandbox.Overlay`, or wherever the consumer wants it) would look like:
```elixir
defmodule MyApp.Overlay do
@moduledoc """
Copy-on-write overlay over any `VFS.Mountable`. Inspect changes with
`diff/1`. Promote with `commit/1` (requires `lower` writable). Discard
by dropping the overlay.
"""
defstruct [:lower, :upper, whiteouts: MapSet.new()]
def new(lower, upper \\ VFS.Memory.new()),
do: %__MODULE__{lower: lower, upper: upper}
def diff(%__MODULE__{} = ov), do: ... # {added, modified, deleted}
def commit(%__MODULE__{} = ov), do: ... # apply diff onto lower
end
defimpl VFS.Mountable, for: MyApp.Overlay do
use VFS.Skeleton
def stream_read(%{upper: u, lower: l, whiteouts: w} = ov, path, opts) do
cond do
MapSet.member?(w, path) -> {:error, VFS.Error.new(:enoent, path: path)}
true ->
case VFS.Mountable.stream_read(u, path, opts) do
{:ok, s, u2} -> {:ok, s, %{ov | upper: u2}}
{:error, %VFS.Error{kind: :enoent}} ->
case VFS.Mountable.stream_read(l, path, opts) do
{:ok, s, l2} -> {:ok, s, %{ov | lower: l2}}
other -> other
end
other -> other
end
end
end
def write_file(%{upper: u} = ov, path, content, opts) do
case VFS.Mountable.write_file(u, path, content, opts) do
{:ok, u2} -> {:ok, %{ov | upper: u2, whiteouts: MapSet.delete(ov.whiteouts, path)}}
err -> err
end
end
def rm(%{whiteouts: w} = ov, path, _opts),
do: {:ok, %{ov | whiteouts: MapSet.put(w, path)}}
# ...readdir merges upper + lower entries minus whiteouts; stat/exists? same
def capabilities(%{upper: u, lower: l}) do
# intersect lower's read-side caps with upper's write-side caps
...
end
end
```
Agent-loop usage (the pattern that motivated the original "layered/CoW" requirement):
```elixir
base_fs = VFS.new()
|> VFS.mount("/repo", some_exgit_repo) # lower: read-only git
|> VFS.mount("/tmp", VFS.Memory.new()) # lower: scratch
sandbox = MyApp.Overlay.new(base_fs)
{result, sandbox} = JustBash.exec(JustBash.new(fs: sandbox), "sed -i 's/foo/bar/' /repo/README.md")
{added, modified, deleted} = MyApp.Overlay.diff(sandbox)
# drop sandbox to discard, or MyApp.Overlay.commit(sandbox) to promote
```
### Read-only wrapper
Even simpler — a one-field wrapper that refuses every mutation. Backends can also self-declare read-only via `capabilities/1`; the wrapper is for cases where you want to *take* a writable backend and *enforce* read-only at the type level for one code path.
```elixir
defmodule MyApp.ReadOnly do
defstruct [:inner]
def new(inner), do: %__MODULE__{inner: inner}
end
defimpl VFS.Mountable, for: MyApp.ReadOnly do
use VFS.Skeleton
# All reads delegate to inner; threading inner through %ReadOnly{}.
def stream_read(%{inner: i} = ro, path, opts) do
case VFS.Mountable.stream_read(i, path, opts) do
{:ok, s, i2} -> {:ok, s, %{ro | inner: i2}}
err -> err
end
end
# ...stat, exists?, readdir, walk all delegate similarly...
# All writes refused regardless of inner's capabilities
def write_file(_, path, _, _), do: {:error, VFS.Error.new(:erofs, path: path)}
def mkdir(_, path, _), do: {:error, VFS.Error.new(:erofs, path: path)}
def rm(_, path, _), do: {:error, VFS.Error.new(:erofs, path: path)}
def capabilities(%{inner: i}),
do: VFS.Mountable.capabilities(i) |> MapSet.intersection(MapSet.new([:read, :native_walk, :native_stream_read, :lazy]))
end
```
---
## Migration deltas
### `just_bash`
Net deletion: `lib/just_bash/fs/backend.ex`, `lib/just_bash/fs/in_memory_fs.ex`, `lib/just_bash/fs/read_only_fs.ex` — gone, replaced by `:vfs` dep.
`lib/just_bash/fs/fs.ex` shrinks to a thin shim or is deleted entirely (callers use `VFS` directly). For a softer migration, keep `JustBash.FS` as `defmodule JustBash.FS, do: defdelegate(..., to: VFS)` for one or two minor releases.
`JustBash.exec/2` already accepts `fs:` in opts. The only behavioural change: it now returns `{result, vfs}` (it already does — see `lib/just_bash.ex`), but the inner `vfs` is now a `%VFS{}` not a `%JustBash.FS{}`.
### `pyex`
`Pyex.Filesystem` (the behaviour) — deleted.
`Pyex.Filesystem.Memory` — deleted (replaced by `VFS.Memory`).
`Pyex.Filesystem.S3` — kept temporarily as a leaf `defimpl VFS.Mountable, for: Pyex.Filesystem.S3` until a stock `VFS.S3` lands. Code-wise, just porting the existing `:read`/`:write` callbacks onto the new protocol surface — no functional change. (The pyex S3 impl is currently the only S3 backend in the stack, and we want to keep that capability working through the migration.)
`Pyex.Ctx`'s `:filesystem` field becomes a `VFS.t()`. `open_handle` / `close_handle` rewrite their `mod = fs.__struct__; mod.read(fs, path)` dispatch to `VFS.Mountable.read_file(fs, path)` — and now thread the returned `fs` back through `ctx`, which they already do.
Two API gaps to reconcile between the two consumer libs:
| Concern | `JustBash.FS.Backend` | `Pyex.Filesystem` | `VFS.Mountable` (proposed) |
|---|---|---|---|
| Content type | `binary()` | `String.t()` | `binary()` (UTF-8 strings are binaries) |
| Errors | `:enoent`, etc. | `"FileNotFoundError: ..."` strings | POSIX atoms (pyex formats Python errors at its boundary) |
| Write modes | `write_file` + `append_file` | `write(:write \| :append \| :read)` | `write_file` + `append_file` (cleaner; `:read` was nonsensical in `write/4`) |
| Symlinks/chmod | yes | no | yes; pyex impls just won't call them |
| Stat shape | bare map with `is_file`/`is_directory` | bare map | `%VFS.Stat{}` (own struct, virtual-FS-shaped); `type` atom replaces booleans |
Pyex's "format Python error string in the FS layer" pattern was wrong — error formatting belongs at the Python-binding boundary, not in the FS. The migration removes it.
---
## What this fixes vs. `feat/mountable-virtual-filesystem-core`
1. **Lazy-backend cache eviction on every read.** `GitFS` had to grow a `materialize/1` workaround precisely because `read_file` couldn't return the cache-updated `Repository.t()`. The new protocol makes the cache survive reads naturally.
2. **Decorators are one-field, not two.** A user-written read-only or CoW wrapper becomes one field (`inner :: any-impl`) where the existing `ReadOnlyFS` has two (`inner_mod`, `inner_state`). Pattern documented in "Worked example patterns" rather than shipped as a stock impl.
3. **Caller-provided is one `defimpl`.** No `__struct__` extraction, no `{module, state}` plumbing; just implement the protocol on your struct.
4. **One library, two consumers.** Right now `pyex` and `just_bash` have completely independent FS abstractions. The new design is the explicit shared dep that lets agents pass FS state between them.
5. **Mount tables nest** because `VFS` itself implements `VFS.Mountable`. Free composition.
The mount-routing logic, longest-prefix matching, synthetic mountpoint stat/readdir merging, cross-mount `mv` returning `:exdev`, symlink-crosses-mount detection — all of that ports over directly. The current implementation in `lib/just_bash/fs/fs.ex` is sound; we're just changing the dispatch substrate underneath it.
---
## Deferred from v1
These were considered and deliberately punted; flagging here so the reviewer doesn't have to ask:
- **`VFS.S3` backend.** When added, will wrap Req's built-in S3 support. Until then, `Pyex.Filesystem.S3` ports forward as a leaf `defimpl VFS.Mountable, for: Pyex.Filesystem.S3` so we don't lose S3 capability during migration.
- **Streaming writes (`stream_write/3`).** Not needed for the agent loop in v1. Would land alongside `VFS.S3`'s multipart upload — at that point the protocol grows one callback with a default `Enum.into` impl.
- **`tail`-style path watching.** Different concern from VFS streaming; would get its own protocol (`VFS.Watch`) when a real workload demands it.
- **Concurrent walk across mounts.** The `%VFS{}` mount-table `walk` concatenates per-mount streams sequentially in v1. Parallelizing is a `Task.async_stream` away when a workload shows it matters.
- **Read/write protocol split.** Single `VFS.Mountable` protocol with `:erofs`/`:enotsup` returns and `capabilities/1` introspection, not separate `Read`/`Write` protocols. The split would force every dispatcher op to pick which protocol to dispatch to, and we'd lose protocol consolidation wins.
## v1 deliverable checklist
**`ivarvong/vfs`:**
- [ ] Repo created, `mix new` skeleton, zero non-stdlib deps
- [ ] `VFS.Stat`, `VFS.Path` (pure path utilities)
- [ ] `VFS.Mountable` protocol — 10 callbacks; Skeleton supplies `walk`/`materialize` defaults
- [ ] `VFS.Default` — fallback impl for `walk`
- [ ] `VFS.Skeleton` — `use`-able macro that wires the defaults
- [ ] `VFS.Memory` — in-memory backend (port + simplification of `JustBash.FS.InMemoryFS`)
- [ ] `%VFS{}` mount-table struct + `defimpl VFS.Mountable, for: VFS` (port of existing routing logic from `feat/mountable-virtual-filesystem-core`)
- [ ] ~~`grep` and `glob` helpers~~ — cut from the library; consumer-side compositions (see "Worked example 1")
- [ ] Conformance test suite parametrized over impls — every backend runs the same test set
- [ ] README documenting the worked-example patterns (CoW overlay, read-only wrapper) so users know how to compose
**`ivarvong/exgit`:**
- [ ] Add `:vfs` as an optional dep
- [ ] `defimpl VFS.Mountable, for: Exgit.Repository` — wraps `Exgit.FS` with native pushdowns for `walk`/`materialize`
- [ ] Tests confirming the defimpl passes vfs's conformance suite (read-only subset)
**`elixir-ai-tools/just_bash`:**
- [ ] PR deleting `lib/just_bash/fs/{backend,in_memory_fs,read_only_fs}.ex` and the `feat/mountable-virtual-filesystem-core` proposal
- [ ] Add `:vfs` dep; `JustBash.exec/2`'s `:fs` opt becomes a `VFS.t()`
- [ ] `JustBash.FS` shim module aliasing to `VFS` for one minor release if needed for migration smoothness
**`ivarvong/pyex`:**
- [ ] PR deleting `Pyex.Filesystem` behaviour and `Pyex.Filesystem.Memory` impl
- [ ] `Pyex.Filesystem.S3` ported as `defimpl VFS.Mountable, for: Pyex.Filesystem.S3` (leaf-only; no behaviour)
- [ ] `Pyex.Ctx`'s `:filesystem` field becomes a `VFS.t()`; ctx threads it through
**Integration:**
- [ ] End-to-end test exercising the agent loop — bash writes via `VFS`, pyex reads via `VFS`, FS state threaded through both — over (a) `VFS.Memory`, (b) `%VFS{}` with a memory mount + an `Exgit.Repository` mount