Skip to main content

CHANGELOG.md

# Changelog

## Unreleased

### Added

- Union underlying types: `union U : int32 { ... }` (any integral
  scalar or a previously declared enum). Discriminators are read and
  written at the underlying type's full width, matching what `flatc`'s
  generated C++/TS code does on the wire. Note that `flatc`'s own JSON
  tooling does not implement the feature (it refuses such schemas for
  `--json` and mis-writes 1-byte discriminators for `--binary`), so
  text-level interop with `flatc` is not possible for these schemas.
- Property-based round-trip tests (StreamData): generators derived
  from the schema IR produce arbitrary valid values and round-trip
  them through encode/decode/verify and through `flatc`.
- Generated root tables expose `file_extension/0` when the schema
  declares `file_extension`, parallel to `file_identifier/0`.
- `verify/2` and `verify_size_prefixed/2` accept `max_depth:`
  (default 64) to bound recursion into nested tables.
- Encode-direction differential suite: buffers produced by the
  generated encoders are now verified against `flatc` across the
  feature matrix (scalars, strings, vectors, structs, fixed arrays,
  enums/bit_flags, unions incl. vectors of unions, nesting,
  file identifiers, size prefixes, key sorting, shared strings,
  force_align). Previously only the decode direction was covered.
- Schema validation in the resolver, mirroring `flatc` semantics:
  duplicate field/variant names, explicit `(id: N)` rules
  (all-or-none, consecutive from 0, unions take two ids), default
  value typing and range checks, enum value ranges (incl. `bit_flags`
  shifts), `required` rejected on scalar fields, `force_align`
  power-of-two/bounds checks, fixed arrays restricted to structs,
  union variant cap (255), empty structs and struct defaults
  rejected, nested vectors rejected. Errors carry the offending name
  and source line.
- Lexer: `\b`, `\f`, `\/`, `\uXXXX` (with surrogate pairs) and
  `\xHH` string escapes; `.5`/`1.` float literals; malformed or
  non-UTF-8 input returns tagged errors instead of raising.
- `root_type` now resolves with the same lookup rules `flatc` uses
  (name-as-written first, then current-namespace-qualified).
- GitHub Actions CI: lint, offline test, full-corpus test (fixtures
  and `flatc` cached), and dialyzer jobs.
- `Mix.Tasks.Compile.Flatbuf` — a Mix compiler that regenerates `.ex`
  files from configured `.fbs` schemas on every `mix compile`. Wire
  it in with `compilers: [:flatbuf | Mix.compilers()]` and configure
  schemas under `config :my_app, :flatbuf`. A per-build manifest
  gates writes (unchanged outputs aren't rewritten) and drops stale
  outputs when schemas are removed from config.
- `mix flatbuf.gen.check` — CI gate that exits non-zero if running
  the gen pipeline would change any committed file.
- `Flatbuf.Table` behaviour and `--niceties` opt-in for generated
  root tables. `--niceties behaviour` attaches the behaviour;
  `--niceties jason` derives `Jason.Encoder` on the struct (the
  consumer must depend on `:jason`). Default remains dependency-free.
- File identifier emission. When a schema declares
  `file_identifier "XXXX";`, the root encode helpers write the
  4-byte marker into the buffer header, and `file_identifier/0`
  surfaces the value on the generated module.
- Size-prefixed buffer support. Every root table now exposes
  `encode_size_prefixed/1`, `decode_size_prefixed/1`, and
  `verify_size_prefixed/1`.
- Required-field enforcement. Fields marked `(required)` in the
  schema cause `encode/1` to fail with
  `{:error, {:flatbuf_required, :name}}` when missing, and `verify/1`
  to fail with `{:error, {:missing_required, :name}}` when the
  buffer's vtable doesn't list the slot.

### Changed

- **Breaking (generated code):** verifier errors are now three-element
  tuples `{:error, reason, path}` where `path` is a root-first list of
  field atoms, vector indices, and union variant atoms locating the
  failure (e.g. `[:inventory, 3, :name]`); buffer-level failures carry
  `[]`. Reasons keep their tagged shapes; field identity that the path
  now carries was dropped from `:union_vector_*` reasons. Regenerate
  committed output and update any `{:error, _}` matches on `verify`
  results.
- Deprecated fields are skipped on encode (their vtable slots stay
  reserved), matching `flatc`'s generated builders — previously they
  were written. Decode still surfaces a deprecated field's value when
  a buffer contains it; `to_json/1` omits deprecated fields.
- Codegen threads the namespace override explicitly instead of via
  the process dictionary, and the wire template uses a validating
  multi-hole fill (internal; emitted sources are byte-identical).
- `Flatbuf.Codegen.generate/2` accepts a `:niceties` option.
- The generated `encode/1` now catches the required-field throw and
  returns it as a tagged error tuple instead of crashing the caller.
- Generated sources are piped through `Code.format_string!/1` before
  being returned/written, so emitted files pass
  `mix format --check-formatted` as-is. Regenerating previously
  committed output will produce a one-time formatting diff.
- The generated `decode/1` no longer catches every exception. It
  rescues only buffer-shaped failures (`MatchError`, `ArgumentError`,
  `FunctionClauseError`) and returns
  `{:error, {:malformed_buffer, exception}}`; other exceptions
  propagate. Use `verify/1` first on untrusted input.

### Fixed

- The `alignment_test` upstream fixture round-trips: the harness had
  been pairing the binary with a stale orphaned JSON from an older
  schema revision. The two remaining pinned fixtures are annotated
  with one-line reasons (`evolution_v1`: `flatc` segfaults generating
  text for its own buffer; `test_64bit`: 64-bit offsets unsupported
  and `flatc` has no JSON oracle for them), and recorded errors no
  longer embed machine-specific paths.
- Scalar values are validated on the encode path. Out-of-range or
  wrong-typed values used to be silently truncated into the wire
  (`encode(%{a: 70_000})` on a `ushort` produced `4464`); they now
  return `{:error, {:scalar_out_of_range | :invalid_scalar, field,
  kind, value}}`. Covers table fields, vectors, struct members,
  fixed arrays, enum underlying values, and hash fields.
- Optional scalar/enum fields (`= null`) no longer crash `encode/1`
  when absent, and explicitly-passed type-default values (`0`,
  `false`, first enum variant) are now written instead of dropped —
  absence and explicit defaults are distinguishable on the wire.
- The verifier bounds-checks inline fields (scalars, enums, inline
  structs, union discriminators) against the table's inline area. A
  crafted vtable could previously direct reads past the buffer while
  `verify/1` returned `:ok`.
- Vector-of-union verification checks both parallel vectors,
  requires equal element counts
  (`{:error, {:union_vector_count_mismatch, ...}}`), and no longer
  raises on count-inflated buffers.
- `nil` (NONE) elements in vectors of unions encode as
  discriminator 0 / offset 0 (the layout `flatc`'s binary layer
  accepts) instead of raising `ArithmeticError`; decode yields `nil`.
- A table whose only depth-recursing field is a vector of unions
  generated a `__verify_at__/3` that failed to compile.
- `:nstandard` was an unconditional runtime dependency; it is now
  dev/test-only, so consumers pull no transitive deps from
  `:flatbuf`.
- Unknown `--niceties` values (e.g. `behavior`) raised silently no
  effect; they are now rejected with the valid values listed.
- `Mix.Tasks.Compile.Flatbuf` tolerates a corrupted manifest,
  reports a missing `:schemas` config key with a friendly message,
  and no longer rewrites byte-identical artifacts (and their mtimes)
  when an unrelated schema changes.
- A fresh clone's `mix test` is green: corpus-gated smoke tests skip
  (instead of failing) with a notice pointing at
  `mix flatbuf.fetch_fixtures` / `mix flatbuf.fetch_flatc`.
- Codegen crashed with a `CaseClauseError` on enum types declared
  with no variants (e.g. `enum Foo : int {}`).

## v0.1.0

- Initial extraction from `sprawl/arrow`: parser, resolver, codegen for
  tables/structs/enums/unions, JSON converter, verifier, and the
  `mix flatbuf.gen` task with namespace/include overrides.