# Precompilation guide
Rustler provides an easy way to use safer NIFs in OTP applications. But in some
environments it's harder to use the benefits of the tool because every user
needs to install the Rust toolchain and compile the project,
which can take several minutes in some cases.
This changes with the help of the `RustlerPrecompiled` package. Now we can easily
use precompiled Rustler NIFs from an external source.
The precompilation happens in a CI server, always in a transparent way, and
the Hex package published should always include a checksum file to ensure
the NIFs stays the same, therefore avoiding supply chain attacks.
In this guide I will show you how to prepare your project to use this feature.
## Prepare for the build
Most of the work is done in the CI server. In this example we are going to use GitHub Actions.
The GH Actions service has the benefit of hosting artifacts for releases and make them
public available.
### Configure Github Actions
In order for the workflow to succeed, read and write permissions will need to be enabled for the
repository.
1. Settings > Actions > General
2. Workflow permissions
3. Check the box "Read and write permissions"
### Configure Targets
Usually we want to build for the most popular targets and the minimum NIF version supported.
NIF versions are more stable than OTP versions because they usually change only after two major
releases of OTP. But older versions are compatible with newer versions if they have the same MAJOR
number. For example, the NIF `2.15` is compatible with `2.16` and `2.17`. So you only need to
compile for `2.15` if you want to support these versions. But in case any new feature from the
newer versions is needed, then you can build for both versions as well.
In Rustler - starting from v0.29 -, it's possible to control which version of NIF is active by
configuring cargo features that have this format: `nif_version_MAJOR_MINOR`. So it's possible
to define features in your project that depends on Rustler features.
More details are in the "Additional configuration before build".
For this guide our targets will be the following:
- OS: Linux, Windows, macOS
- Architectures: `x86_64`, `aarch64` (ARM 64 bits), `arm`
- NIF versions: `2.15`, `2.16`.
In summary the build matrix looks like this:
```yaml
matrix:
nif: ["2.16", "2.15"]
job:
- { target: arm-unknown-linux-gnueabihf , os: ubuntu-20.04 , use-cross: true }
- { target: aarch64-unknown-linux-gnu , os: ubuntu-20.04 , use-cross: true }
- { target: aarch64-apple-darwin , os: macos-11 }
- { target: x86_64-apple-darwin , os: macos-11 }
- { target: x86_64-unknown-linux-gnu , os: ubuntu-20.04 }
- { target: x86_64-unknown-linux-musl , os: ubuntu-20.04 , use-cross: true }
- { target: x86_64-pc-windows-gnu , os: windows-2019 }
- { target: x86_64-pc-windows-msvc , os: windows-2019 }
```
A complete workflow example can be found in the [`rustler_precompilation_example`](https://github.com/philss/rustler_precompilation_example/blob/main/.github/workflows/release.yml) project.
That workflow is using a GitHub Action especially made for our goal: [philss/rustler-precompiled-action](https://github.com/philss/rustler-precompiled-action).
The GitHub Action will deal with the installation of `cross` and the build of the project, naming the files in the correct format.
Some targets are only supported by later versions of `cross`. For those, you might want to
install `cross` directly from GitHub. You can see an example in [this
pipeline](https://github.com/kloeckner-i/mail_parser/blob/f4af5083aec73a47f0e41a202ba46a91f60602cf/.github/workflows/release.yml#L101-L105).
## Additional configuration before build
In our build we are going to cross compile our crate project (the Rust code for our NIF) using
a variety of targets, as we saw in the previous section. For this to work we need to guide the Rust
compiler in some cases by providing additional configuration in the `.cargo/config` file of our project.
Here is an example of that file:
```toml
[target.'cfg(target_os = "macos")']
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
# See https://github.com/rust-lang/rust/issues/59302
[target.x86_64-unknown-linux-musl]
rustflags = [
"-C", "target-feature=-crt-static"
]
# Provides a small build size, but takes more time to build.
[profile.release]
lto = true
```
In addition to that, we also use a tool called [`cross`](https://github.com/rust-embedded/cross) that
makes the build easier for some targets (the ones using `use-cross: true` in our example).
For projects using Rustler **before v0.29**, we need to tell `cross` to read an environment variable
from our "host machine", because `cross` uses containers to build our software.
So you need to create the file `Cross.toml` in the NIF directory with the following content:
```toml
[build.env]
passthrough = [
"RUSTLER_NIF_VERSION"
]
```
#### Using features to control NIF version in Rustler v0.29 and above
Since Rustler v0.29, it's possible to control which NIF version is active by using cargo features.
This is a replacement for the `RUSTLER_NIF_VERSION` env var, that is deprecated in v0.30 of
Rustler.
If your project does not use anything special from newer NIF versions, then you can declare the
Rustler dependency like this:
```toml
[dependencies]
rustler = { version = "0.29", default-features = false, features = ["derive", "nif_version_2_15"] }
```
And in the workflow file, you would specify the `nif-version: 2.15` as usual.
But in case you want to have newer features from more recent versions of NIF, you can create
features for your project that are used to activate rustler features. These features should
follow the same naming from Rustler, because the CI action is going to use that to activate
the right feature.
Here is an example of how your `Cargo.toml` would look like:
```toml
[dependencies]
rustler = { version = "0.29", default-features = false, features = ["derive"] }
# And then, your features.
[features]
default = ["nif_version_2_15"]
nif_version_2_15 = ["rustler/nif_version_2_15"]
nif_version_2_16 = ["rustler/nif_version_2_16"]
nif_version_2_17 = ["rustler/nif_version_2_17"]
```
In your code, you would use these features - like `nif_version_2_17` - to control how your
code is going to be compiled. You can hide some features behind these features.
Even if you don't have anything behind these features, you can still introduce them
if you want to activate an specific NIF version.
But again, normally it's enough to build for the lowest version supported by the OTP version
that you are targeting.
The available NIF versions are the following:
* `2.14` - for OTP 21 and above.
* `2.15` - for OTP 22 and above.
* `2.16` - for OTP 24 and above.
* `2.17` - for OTP 26 and above.
## The Rustler module
We need to tell `RustlerPrecompiled` where to find our NIF files, and we need to tell which version to use.
```elixir
defmodule RustlerPrecompilationExample.Native do
version = Mix.Project.config()[:version]
use RustlerPrecompiled,
otp_app: :rustler_precompilation_example,
crate: "example",
base_url:
"https://github.com/philss/rustler_precompilation_example/releases/download/v#{version}",
force_build: System.get_env("RUSTLER_PRECOMPILATION_EXAMPLE_BUILD") in ["1", "true"],
version: version
# When your NIF is loaded, it will override this function.
def add(_a, _b), do: :erlang.nif_error(:nif_not_loaded)
end
```
This example was extracted from the [`rustler_precompilation_example`](https://github.com/philss/rustler_precompilation_example/blob/main/lib/rustler_precompilation_example/native.ex) project.
RustlerPrecompiled will try to figure out the target and download the correct file for us. This will happen in compile
time only.
Optionally it's possible to force the compilation by setting an env var, like the example suggests.
It's also possible to force the build by using a pre release version, like `0.1.0-dev`. The only
requirement to force the build is to have Rustler declared as a dependency as well:
`{:rustler, ">= 0.0.0", optional: true}`.
## The release flow
### Generating a checksum file
In a scenario where you need to release a Hex package using precompiled NIFs, you first need to
build the release in the CI, wait for all artifacts to be available and then generate
the **checksum file** that is **MANDATORY** for your package to work.
This checksum file is generated by running the following command after the build is complete:
$ mix rustler_precompiled.download YourRustlerModule --all --print
With the module I used for this guide, the command would be:
$ mix rustler_precompiled.download RustlerPrecompilationExample.Native --all --print
The file generated will be named `checksum-Elixir.RustlerPrecompilationExample.Native.exs` and
it's **extremely important that you include this file in your Hex package** (by updating the `files:`
field in your `mix.exs`). Otherwise your package **won't work**. Your `files:` key at your
package configuration will look like this:
```elixir
defp package do
[
files: [
"lib",
"native/example/.cargo",
"native/example/src",
"native/example/Cargo*",
"checksum-*.exs",
"mix.exs"
],
# ...
]
end
```
Note: you don't need to track the checksum file in your version control system (git or other).
For an example, refer to the `mix.exs` file of the [rustler precompilation example](https://github.com/philss/rustler_precompilation_example/blob/main/mix.exs)
or elixir-nx's [explorer](https://github.com/elixir-nx/explorer/blob/723eea63204e43bc9238d2488fd355f17a1e13f2/mix.exs#L65-L72) library.
Tip: use the `mix hex.build --unpack` command to confirm which files are being included (and if the package looks good before publishing).
### Recommended flow
To recap, the suggested flow is the following:
1. release a new tag
2. push the code to your repository with the new tag: `git push origin main --tags`
3. wait for all NIFs to be built
4. run the `mix rustler_precompiled.download` task (with the flag `--all`)
5. release the package to Hex.pm (make sure your release includes the correct files).
## Conclusion
The ability to use precompiled NIFs written in Rust can increase the adoption of some packages,
because people won't need to have Rust installed. But this comes with some drawbacks and more
responsibilities to the maintainers, so use this feature carefully.
