# Domo
## Section
|[](https://travis-ci.com/IvanRublev/domo)|[](#domo)|[](https://hex.pm/packages/domo)|
|-|-|-|
⚠️ This library generates code for structures that can bring suboptimal compilation times increased to approximately 20%
🔗 JSON parsing and validation example is in https://github.com/IvanRublev/contentful-elixir-parse-example-nestru-domo repo.
🔗 Commanded + Domo combo used in Event Sourcing and CQRS example app is in https://github.com/IvanRublev/bank-commanded-domo repo.
---
<!-- Documentation -->
A library to ensure the consistency of structs modelling a business domain via
their `t()` types and associated precondition functions.
Used in a struct's module, the library adds constructor, validation,
and reflection functions. Constructor and validation functions
guarantee the following at call time:
* A complex struct conforms to its `t()` type.
* Structs are validated to be consistent to follow given business rules by
precondition functions associated with struct types.
If the conditions described above are not met, the constructor
and validation functions return an error.
Because precondition function associates with type the validation can be shared
across all structs referencing the type.
In terms of Domain Driven Design the invariants relating structs to each other
can be defined with types and associated precondition functions.
---
<p align="center">
<a href="https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2FIvanRublev%2FDomo%2Fblob%2Fmaster%2FREADME.md">
<img src="https://livebook.dev/badge/v1/blue.svg" alt="Run in Livebook" />
</a>
</p>
```elixir
Mix.install([:domo], force: true)
```
---
Let's say that we have a `LineItem` and `PurchaseOrder` structs with relating
invariant that is the sum of line item amounts should be less then order's
approved limit. That can be expressed like the following:
```elixir
defmodule LineItem do
use Domo
defstruct amount: 0
@type t :: %__MODULE__{amount: non_neg_integer()}
end
defmodule PurchaseOrder do
use Domo
defstruct id: 1000, approved_limit: 200, items: []
@type id :: non_neg_integer()
precond(id: &(1000 <= &1 and &1 <= 5000))
@type t :: %__MODULE__{
id: id(),
approved_limit: pos_integer(),
items: [LineItem.t()]
}
precond(t: &validate_invariants/1)
defp validate_invariants(po) do
amounts = po.items |> Enum.map(& &1.amount) |> Enum.sum()
if amounts <= po.approved_limit do
:ok
else
{:error, "Sum of line item amounts (#{amounts}) should be <= to approved limit (#{po.approved_limit})."}
end
end
end
```
Then `PurchaseOrder` struct can be constructed consistently with functions generated by Domo like the following:
```elixir
PurchaseOrder.new()
```
```output
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: []}}
```
The constructor function takes any `Enumerable` as the input value:
```elixir
{:ok, po} = PurchaseOrder.new(%{approved_limit: 250})
```
```output
{:ok, %PurchaseOrder{approved_limit: 250, id: 1000, items: []}}
```
It returns the descriptive keyword list if there is an error in input arguments.
And it validates nested structs automatically:
```elixir
PurchaseOrder.new(id: 500, items: [%LineItem{amount: -5}])
```
```output
{:error,
[
items: "Invalid value [%LineItem{amount: -5}] for field :items of %PurchaseOrder{}.
Expected the value matching the [%LineItem{}] type.
Underlying errors:
- The element at index 0 has value %LineItem{amount: -5} that is invalid.
- Value of field :amount is invalid due to Invalid value -5 for field :amount
of %LineItem{}. Expected the value matching the non_neg_integer() type.",
id: "Invalid value 500 for field :id of %PurchaseOrder{}. Expected the
value matching the non_neg_integer() type. And a true value from
the precondition function \"&(1000 <= &1 and &1 <= 5000)\"
defined for PurchaseOrder.id() type."
]}
```
And manually updated struct can be validated like the following:
```elixir
po
|> Map.put(:items, [LineItem.new!(amount: 150)])
|> PurchaseOrder.ensure_type()
```
```output
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: [%LineItem{amount: 150}]}}
```
Domo returns the error if the precondition function attached to the `t()` type
that validates invariants for the struct as a whole fails:
```elixir
updated_po = %{po | items: [LineItem.new!(amount: 180), LineItem.new!(amount: 100)]}
PurchaseOrder.ensure_type(updated_po)
```
```output
{:error, [t: "Sum of line item amounts should be <= to approved limit"]}
```
Getting the list of the required fields of the struct that have type other
then `nil` or `any` is like that:
```elixir
PurchaseOrder.required_fields()
```
```output
[:approved_limit, :id, :items]
```
See the [Callbacks](#callbacks) section for more details about functions added to the struct.
## Integration with Ecto
Ecto schema changeset can be automatically validated to conform to `t()` type
and associated preconditions. Then the changeset function can be like the following:
<!-- livebook:{"force_markdown":true} -->
```elixir
defmodule Customer do
use Ecto.Schema
use Domo, ensure_struct_defaults: false
import Ecto.Changeset
import Domo.Changeset
schema "customers" do
field :first_name, :string
field :last_name, :string
field :birth_date, :date
timestamps()
end
@type t :: %__MODULE__{
first_name: String.t(),
last_name: String.t(),
birth_date: Date.t()
}
def changeset(changeset, attrs) do
changeset
|> cast(attrs, typed_fields())
|> validate_required(required_fields())
|> validate_type()
end
# Domo adds typed_fields/0, required_fields/0 funcitons to the schema.
# Domo.Cahngeset defines validate_type/1 function.
end
```
See `typed_fields/0`, `required_fields/0`, and `Domo.Changeset` module
documentation for details.
See detailed example is in the [./example_avialia](https://github.com/IvanRublev/Domo/tree/master/example_avialia) project.
## Integration with libraries generating t() type for a struct
Domo is compatible with most libraries that generate `t()` type for a struct
or an Ecto schema. Just `use Domo` in the module, and that's it.
An advanced example is in the [./example_typed_integrations](https://github.com/IvanRublev/Domo/tree/master/example_typed_integrations) project.
## Compile-time and Run-time validations
At the project's compile-time, Domo can perform the following checks:
* It automatically validates that the default values given with `defstruct/1`
conform to struct's type and fulfill preconditions.
* It ensures that the struct using Domo built with `new!/1` function
to be a function's default argument or a struct field's default value
matches its type and preconditions.
Domo validates struct type conformance with appropriate `TypeEnsurer` modules
built during the project's compilation at the application's run-time.
These modules rely on guards and pattern matchings. See `__using__/1` for
more details.
## Depending types tracking
Suppose the given structure field's type depends on a type defined in
another module. When the latter type or its precondition changes,
Domo recompiles the former module automatically to update its
`TypeEnsurer` to keep type validation in current state.
That works similarly for any number of intermediate modules
between module defining the struct's field and module defining the field's final type.
## Setup
To use Domo in a project, add the following line to `mix.exs` dependencies:
<!-- livebook:{"force_markdown":true} -->
```elixir
{:domo, "~> 1.2.0"}
```
And the following line to the compilers:
<!-- livebook:{"force_markdown":true} -->
```elixir
compilers: Mix.compilers() ++ [:domo_compiler]
```
To avoid `mix format` putting extra parentheses around `precond/1` macro call,
add the following import to the `.formatter.exs`:
<!-- livebook:{"force_markdown":true} -->
```elixir
[
import_deps: [:domo]
]
```
## Setup for Phoenix hot reload
To enable hot reload for type changes in structs using Domo, add the following
line to the endpoint's configuration in the `config.exs` file:
<!-- livebook:{"force_markdown":true} -->
```elixir
config :my_app, MyApp.Endpoint,
reloadable_compilers: [:phoenix] ++ Mix.compilers() ++ [:domo_compiler]
```
<!-- Documentation -->
## Callbacks
Constructor, validation, and reflection functions added to the struct module using Domo.
### new!/1/0
<!-- new!/1 -->
Creates a struct validating type conformance and preconditions.
The argument is any `Enumerable` that emits two-element tuples
(key-value pairs) during enumeration.
Returns the instance of the struct built from the given `enumerable`.
Does so only if struct's field values conform to its `t()` type
and all field's type and struct's type precondition functions return ok.
Raises an `ArgumentError` if conditions described above are not fulfilled.
This function will check if every given key-value belongs to the struct
and raise `KeyError` otherwise.
<!-- new!/1 -->
### new/2/1/0
<!-- new/2 -->
Creates a struct validating type conformance and preconditions.
The argument is any `Enumerable` that emits two-element tuples
(key-value pairs) during enumeration.
Returns the instance of the struct built from the given `enumerable`
in the shape of `{:ok, struct_value}`. Does so only if struct's
field values conform to its `t()` type and all field's type and struct's
type precondition functions return ok.
If conditions described above are not fulfilled, the function
returns an appropriate error in the shape of `{:error, message_by_field}`.
`message_by_field` is a keyword list where the key is the name of
the field and value is the string with the error message.
Keys in the `enumerable` that don't exist in the struct
are automatically discarded.
#### Options
* `maybe_filter_precond_errors` - when set to `true`, the values in
`message_by_field` instead of string become a list of error messages
from precondition functions. If there are no error messages from
precondition functions for a field's type, then all errors are returned
unfiltered. Helpful in taking one of the custom errors after executing
precondition functions in a deeply nested type to communicate
back to the user. F.e. when the field's type is another struct.
Default is `false`.
<!-- new/2 -->
### ensure_type!/1
<!-- ensure_type!/1 -->
Ensures that struct conforms to its `t()` type and all preconditions
are fulfilled.
Returns struct when it's valid. Raises an `ArgumentError` otherwise.
Useful for struct validation when its fields changed with map syntax
or with `Map` module functions.
<!-- ensure_type!/1 -->
### ensure_type/2/1
<!-- ensure_type/2 -->
Ensures that struct conforms to its `t()` type and all preconditions
are fulfilled.
Returns struct when it's valid in the shape of `{:ok, struct}`.
Otherwise returns the error in the shape of `{:error, message_by_field}`.
Useful for struct validation when its fields changed with map syntax
or with `Map` module functions.
<!-- ensure_type/2 -->
### typed_fields/1/0
<!-- typed_fields/1 -->
Returns the list of struct's fields defined with explicit types in its `t()` type spec.
Does not return meta fields with `__underscored__` names and fields
having `any()` type by default.
Includes fields that have `nil` type into the return list.
#### Options
* `:include_any_typed` - when set to `true`, adds fields with `any()`
type to the return list. Default is `false`.
* `:include_meta` - when set to `true`, adds fields
with `__underscored__` names to the return list. Default is `false`.
<!-- typed_fields/1 -->
### required_fields/1/0
<!-- required_fields/1 -->
Returns the list of struct's fields having type others then `nil` or `any()`.
Does not return meta fields with `__underscored__` names.
Useful for validation of the required fields for emptiness.
F.e. with `validate_required/2` call in the `Ecto` changeset.
#### Options
* `:include_meta` - when set to `true`, adds fields
with `__underscored__` names to the return list. Default is `false`.
<!-- required_fields/1 -->
## Limitations
The recursive types like `@type t :: :end | {integer, t()}` are not supported.
Because of that types like `Macro.t()` or `Path.t()` are not supported.
Parametrized types are not supported. Library returns `{:type_not_found, :key}`
error for `@type dict(key, value) :: [{key, value}]` type definition.
Domo returns error for type referencing parametrized type like
`@type field :: container(integer())`.
Generated submodule with TypedStruct's `:module` option is not supported.
## Migration
To complete the migration to a new version of Domo, please, clean and recompile
the project with `mix clean --deps && mix compile` command.
## Adoption
It's possible to adopt Domo library in the project having user-defined
constructor functions as the following:
1. Add `:domo` dependency to the project, configure compilers as described in
the [setup](#setup) section
2. Set the name of the Domo generated constructor function by adding
`config :domo, :name_of_new_function, :constructor_name` option into
the `confix.exs` file, to prevent conflict with original constructor
function names if any
3. Add `use Domo` to existing struct
4. Change the calls to build the struct for Domo generated constructor
function with name set on step 3 and remove original constructor function
5. Repeat for each struct in the project
## Performance 🐢
On the average, the current version of the library makes struct operations
about 20% sower what may seem plodding. And it may look like non-performant
to run in production.
It's not that. The library ensures the correctness of data types at runtime and
it comes with the price of computation. As the result users get the application
with correct states at every update that is valid in many business contexts.
Please, find the output of `mix benchmark` command below.
```
Generate 10000 inputs, may take a while.
=========================================
Construction of a struct
=========================================
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.12.3
Erlang 24.0.1
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 14 s
Benchmarking __MODULE__.new!(arg)...
Benchmarking struct!(__MODULE__, arg)...
Name ips average deviation median 99th %
struct!(__MODULE__, arg) 14.09 K 70.96 μs ±63.01% 72 μs 158 μs
__MODULE__.new!(arg) 11.77 K 84.93 μs ±53.72% 87 μs 181 μs
Comparison:
struct!(__MODULE__, arg) 14.09 K
__MODULE__.new!(arg) 11.77 K - 1.20x slower +13.97 μs
A struct's field modification
=========================================
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.12.3
Erlang 24.0.1
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 14 s
Benchmarking %{tweet | user: arg} |> __MODULE__.ensure_type!()...
Benchmarking struct!(tweet, user: arg)...
Name ips average deviation median 99th %
struct!(tweet, user: arg) 15.01 K 66.62 μs ±66.93% 70 μs 148 μs
%{tweet | user: arg} |> __MODULE__.ensure_type!() 13.53 K 73.89 μs ±60.83% 75 μs 159 μs
Comparison:
struct!(tweet, user: arg) 15.01 K
%{tweet | user: arg} |> __MODULE__.ensure_type!() 13.53 K - 1.11x slower +7.27 μs
```
## Contributing
1. Fork the repository and make a feature branch
2. After implementing of the feature format the code with:
```
mix format
```
run linter and tests to ensure that all works as expected with:
```
mix check || mix check --failed
```
3. Make a PR to this repository
## Changelog
### 1.5.0
* Fix bug to return explicit file read error message during the compile time
* Completely replace `apply()` with `.` for validation function calls to run faster
* Link planner server to mix process for better state handling
* Support of the interactive use in `iex` and `live book`
Breaking change:
* Improve compilation speed by starting resolve planner only once in Domo mix task.
To migrate, please, put the `:domo_compiler` before `:elixir` in mix.exs.
And do the same for `reloadable_compilers` key in config file
if configured for Phoenix endpoint.
### 1.4.1
* Improve compatibility with Elixir v1.13
* Format string representations of an anonymous function passed to `precond/1` macro error message
### 1.4.0
* Fix bug to detect runtime mode correctly when launched under test.
* Add support for `@opaque` types.
Breaking changes:
* Change `new_ok` constructor function name to `new` that is more convenient.
Search and replace `new_ok(` -> `new(` in all files of the project
using Domo to migrate.
* Constructor function name generation procedure changes to adding `!`
to the value of `:name_of_new_function` option. The defaults are `new` and `new!`.
### 1.3.4
* Make error messages to be more informative
* Improve compatibility with `Ecto` 3.7.x
* Explicitly define `:ecto` and `:decimal` as optional dependencies
* Fix bug to pass `:remote_types_as_any` option with `use Domo`
* Explicitly define that `MapSet` should be validated with `precond` function for custom user type, because parametrized `t(value)` types are not supported
* Replace `apply()` with Module.function calls to run faster
### 1.3.3
* Support validation of `Decimal.t()`
* Fix bug to define precondition function for user type referencing any() or term()
### 1.3.2
* Support remote types in erlang modules like `:inet.port_number()`
* Shorten the invalid value output in the error message
* Increase validation speed by skipping fields that are not in `t()` type spec or have the `any()` type
* Fix bug to skip validation of struct's enforced keys default value because they are ignored during the construction anyway
* Increase validation speed by generating `TypeEnsurer` modules for `Date`, `Date.Range`, `DateTime`, `File.Stat`, `File.Stream`, `GenEvent.Stream`, `IO.Stream`, `Macro.Env`, `NaiveDateTime`, `Range`, `Regex`, `Task`, `Time`, `URI`, and `Version` structs from the standard library at the first project compilation
* Fix bug to call the `precond` function of the user type pointing to a struct
* Increase validation speed by encouraging to use Domo or to make a `precond` function for struct referenced by a user type
* Add `Domo.has_type_ensurer?/1` that checks whether a `TypeEnsurer` module was generated for the given struct.
* Add example of parsing with validating of the Contentful JSON reply via `Jason` + `ExJSONPath` + `Domo`
### 1.3.1
* Fix bug to validate defaults having | nil type.
### 1.3.0
* Change the default name of the constructor function to `new!` to follow Elixir naming convention.
You can always change the name with the `config :domo, :name_of_new_function, :new_func_name_here` app configuration.
* Fix bug to validate defaults for every required field in a struct except `__underscored__` fields at compile-time.
* Check whether the precondition function associated with `t()` type returns `true` at compile time regarding defaults correctness check.
* Add examples of integrations with `TypedStruct` and `TypedEctoSchema`.
### 1.2.9
* Fix bug to acknowledge that type has been changed after a failed compilation.
* Fix bug to match structs not using Domo with a field of `any()` type with and without precondition.
* Add `typed_fields/1` and `required_fields/1` functions.
* Add `maybe_filter_precond_errors: true` option that filters errors from precondition functions for better output for the user.
### 1.2.8
* Add `Domo.Changeset.validate_type/*` functions to validate Echo.Changeset field changes matching the t() type.
* Fix the bug to return custom error from precondition function as underlying error for :| types.
### 1.2.7
* Fix the bug to make recompilation occur when fixing alias for remote type.
* Support custom errors to be returned from functions defined with `precond/1`.
### 1.2.6
* Validates type conformance of default values given with `defstruct/1` to the
struct's `t()` type at compile-time.
* Includes only the most matching type error into the error message.
### 1.2.5
* Add `remote_types_as_any` option to disable validation of specified complex
remote types. What can be replaced by precondition for wrapping user-defined type.
### 1.2.4
* Speedup resolving of struct types
* Limit the number of allowed fields types combinations to 4096
* Support `Range.t()` and `MapSet.t()`
* Keep type ensurers source code after compiling umbrella project
* Remove preconditions manifest file on `mix clean` command
* List processed structs giving mix `--verbose` option
### 1.2.3
* Support struct's attribute introduced in Elixir 1.12.0 for error checking
* Add user-defined precondition functions to check the allowed range of values
with `precond/1` macro
### 1.2.2
* Add support for `new/1` calls at compile time f.e. to specify default values
### 1.2.1
* Domo compiler is renamed to `:domo_compiler`
* Compile `TypeEnsurer` modules only if struct changes or dependency type changes
* Phoenix hot-reload with `:reloadable_compilers` option is fully supported
### 1.2.0
* Resolve all types at compile time and build `TypeEnsurer` modules for all structs
* Make Domo library work with Elixir 1.11.x and take it as the required minimum version
* Introduce `---/2` operator to make tag chains with `Domo.TaggedTuple` module
### 0.0.x - 1.0.x
* MVP like releases, resolving types at runtime. Adds `new` constructor to a struct
## Roadmap
## Roadmap
* [x] Check if the field values passed as an argument to the `new/1`, and `put/3` matches the field types defined in `typedstruct/1`.
* [x] Support the keyword list as a possible argument for the `new/1`.
* [x] Add module option to put a warning in the console instead of raising of the `ArgumentError` exception on value type mismatch.
* [x] Make global environment configuration options to turn errors into warnings that are equivalent to module ones.
* [x] Move type resolving to the compile time.
* [x] Keep only bare minimum of generated functions that are `new/1`, `ensure_type!/1` and their _ok versions.
* [x] Make the `new/1` and `ensure_type!/1` speed to be less or equal to 1.5 times of the `struct!/2` speed.
* [x] Support `new/1` calls in macros to specify default values f.e. in other structures. That is to check if default value matches type at compile time.
* [x] Support `precond/1` macro to specify a struct field value's contract with a boolean function.
* [ ] Support types referencing itself for tree structures.
* [ ] Evaluate full recompilation time for 1000 structs using Domo.
* [x] Add use option to specify names of the generated functions.
* [x] Add documentation to the generated for `new(_ok)/1`, and `ensure_type!(_ok)/1` functions in a struct.
## License
Copyright © 2021 Ivan Rublev
This project is licensed under the [MIT license](LICENSE).
