# MDEx
<!-- MDOC -->
<div align="center">
<img src="https://raw.githubusercontent.com/leandrocp/mdex/main/assets/images/mdex_logo.png" width="360" alt="MDEx logo" />
<br>
<a href="https://hex.pm/packages/mdex">
<img alt="Hex Version" src="https://img.shields.io/hexpm/v/mdex">
</a>
<a href="https://hexdocs.pm/mdex">
<img alt="Hex Docs" src="https://img.shields.io/badge/hex.pm-docs-green.svg?style=flat">
</a>
<a href="https://opensource.org/licenses/MIT">
<img alt="MIT" src="https://img.shields.io/hexpm/l/mdex">
</a>
<p align="center">Fast and Extensible Markdown for Elixir.</p>
</div>
## Features
- Fast
- Compliant with the [CommonMark spec](https://commonmark.org)
- [Plugins](https://hexdocs.pm/mdex/plugins.html)
- Formats:
- Markdown (CommonMark)
- HTML
- [Phoenix HEEx](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.Rendered.html)
- JSON
- XML
- [Quill Delta](https://quilljs.com/docs/delta/)
- [Slack mrkdwn](https://docs.slack.dev/messaging/formatting-message-text)
- Floki-like [Document AST](https://hexdocs.pm/mdex/MDEx.Document.html)
- Req-like [Document pipeline API](https://hexdocs.pm/mdex/MDEx.Document.html)
- [GitHub Flavored Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)
- [GitLab Flavored Markdown](https://docs.gitlab.com/user/markdown)
- Discord Flavored Markdown (Partial)
- Wiki-style links
- Phoenix HEEx components and expressions
- [Streaming](https://hexdocs.pm/mdex/streaming.html) incomplete fragments
- [Emoji](https://www.webfx.com/tools/emoji-cheat-sheet) shortcodes
- Built-in Syntax Highlighting with [Lumis](https://mdex.hexdocs.pm/lumis.html) or [Syntect](https://mdex.hexdocs.pm/syntect.html)
- [Code Block Decorators](https://hexdocs.pm/mdex/code_block_decorators-2.html)
- HTML sanitization
- [~MD Sigil](https://hexdocs.pm/mdex/MDEx.Sigil.html) for Markdown, HTML, HEEx, JSON, XML, and Quill Delta
## Plugins
- [mdex_gfm](https://hex.pm/packages/mdex_gfm) - Enable [GitHub Flavored Markdown](https://github.github.com/gfm) (GFM)
- [mdex_mermaid](https://hex.pm/packages/mdex_mermaid) - Render [Mermaid](https://mermaid.js.org) diagrams in code blocks
- [mdex_katex](https://hex.pm/packages/mdex_katex) - Render math formulas using [KaTeX](https://katex.org)
- [mdex_video_embed](https://hex.pm/packages/mdex_video_embed) - Privacy-respecting video embeds from code blocks
- [mdex_custom_heading_id](https://hex.pm/packages/mdex_custom_heading_id) - Custom heading IDs for markdown headings
- [mdex_mermex](https://hex.pm/packages/mdex_mermex) - Render [Mermaid](https://mermaid.js.org) diagrams server-side using Mermex (Rust NIF)
## Installation
Add `:mdex` dependency:
```elixir
def deps do
[
{:mdex, "~> 0.12"}
]
end
```
Or use [Igniter](https://hexdocs.pm/igniter):
```sh
mix igniter.install mdex
```
## Usage
#### Convert to HTML
```elixir
iex> MDEx.to_html!("# Hello :smile:", extension: [shortcodes: true])
"<h1>Hello 😄</h1>"
```
#### Syntax Highlighting
Syntax highlight code blocks using either [Lumis](https://mdex.hexdocs.pm/lumis.html) or [Syntect](https://mdex.hexdocs.pm/syntect.html),
for example to use Lumis:
```elixir
def deps do
[
{:mdex, "~> 0.12"},
{:lumis, "~> 0.1"}
]
end
```
```elixir
config :mdex_native, syntax_highlighter: :lumis
```
#### GitHub Flavored Markdown (GFM)
_Using the `:html_multi_themes` syntax highlighting formatter is not required but you get light/dark using it._
````elixir
Mix.install([
{:mdex_gfm, "~> 0.1"}
])
markdown = """
- [x] Set up project
- [ ] Write docs
```elixir
spawn(fn -> send(current, {self(), 1 + 2}) end)
```
"""
MDEx.new(
markdown: markdown,
syntax_highlight: [
engine: :lumis,
opts: [
formatter: {:html_multi_themes,
themes: [light: "github_light", dark: "github_dark"],
default_theme: "light-dark()"}
]
]
)
|> MDExGFM.attach()
|> MDEx.to_html!()
````
#### Sigils
```elixir
iex> import MDEx.Sigil
iex> ~MD[# Hello :smile:]HTML
"<h1>Hello 😄</h1>"
```
```elixir
iex> import MDEx.Sigil
iex> assigns = %{project: "MDEx"}
iex> ~MD[# {@project}]HEEX
%Phoenix.LiveView.Rendered{...}
```
```elixir
iex> import MDEx.Sigil
iex> ~MD[# Hello :smile:]
#MDEx.Document(3 nodes)<
├── 1 [heading] level: 1, setext: false
│ ├── 2 [text] literal: "Hello "
│ └── 3 [short_code] code: "smile", emoji: "😄"
>
```
#### Streaming
```elixir
iex> MDEx.new(streaming: true)
...> |> MDEx.Document.put_markdown("**Install")
...> |> MDEx.to_html!()
"<p><strong>Install</strong></p>"
```
## Examples and Guides
In docs you can find [Livebook examples](https://hexdocs.pm/mdex/gfm.html) covering options and usage, and [Guides](https://hexdocs.pm/mdex/plugins.html) for more info.
## Foundation
The library is built on top of:
- [comrak](https://crates.io/crates/comrak) - a fast Rust port of [GitHub's CommonMark parser](https://github.com/github/cmark-gfm)
- [ammonia](https://crates.io/crates/ammonia) for HTML Sanitization
- [lumis](https://crates.io/crates/lumis) for Syntax Highlighting
<!-- MDOC -->
## Used By
- [00](https://github.com/technomancy-dev/00)
- [Algora](https://github.com/algora-io/algora)
- [Ash AI](https://github.com/ash-project/ash_ai)
- [BeaconCMS](https://github.com/BeaconCMS/beacon)
- [BeamLens Web](https://github.com/beamlens/beamlens_web)
- [Bonfire](https://github.com/bonfire-networks/bonfire-app)
- [Canada Navigator](https://github.com/canada-ca/navigator)
- [Conpipe](https://github.com/andyl/conpipe)
- [Exmeralda](https://github.com/bitcrowd/exmeralda)
- [Hexpm](https://github.com/hexpm/hexpm)
- [Jido](https://github.com/agentjido/jido)
- [Loomkin](https://github.com/bleuropa/loomkin)
- [Phoenix SEO](https://github.com/dbernheisel/phoenix_seo)
- [Phoenix Storybook](https://github.com/phenixdigital/phoenix_storybook)
- [Plural Console](https://github.com/pluralsh/console)
- [Prosody](https://github.com/halostatue/prosody)
- [Reposit](https://github.com/reposit-bot/reposit)
- [Sagents Live Debugger](https://github.com/sagents-ai/sagents_live_debugger)
- [Sayfa](https://github.com/furkanural/sayfa)
- [Tableau](https://github.com/elixir-tools/tableau)
- [Teiserver](https://github.com/beyond-all-reason/teiserver)
- [TermUI](https://github.com/pcharbon70/term_ui)
- [Tuist](https://github.com/tuist/tuist)
- [Valentine](https://github.com/maxneuvians/valentine)
- [Wraft](https://github.com/wraft/wraft)
- And [more...](https://github.com/search?q=lang%3Aelixir+%3Amdex&type=code)
_Are you using MDEx and want to list your project here? Please send a PR!_
## Sponsors
💜 **Support MDEx Development**
If you or your company find MDEx useful, please consider sponsoring its development.
➡️ **[GitHub Sponsors](https://github.com/sponsors/leandrocp)**
Your support helps maintain and improve MDEx for the entire Elixir community!
**Current and previous sponsors**
<a href="https://dockyard.com" target="_blank"><img src="assets/images/dockyard_logo.svg" width="200" alt="DockYard" /></a>
## Motivation
MDEx was born out of the necessity of parsing CommonMark files, to parse hundreds of files quickly, and to be easily extensible by consumers of the library.
- [earmark](https://hex.pm/packages/earmark) is extensible but [can't parse](https://github.com/RobertDober/earmark_parser/issues/126) all kinds of documents and is slow to convert hundreds of markdowns.
- [md](https://hex.pm/packages/md) is very extensible but the doc says "If one needs to perfectly parse the common markdown, Md is probably not the correct choice" and CommonMark was a requirement to parse many existing files.
- [markdown](https://hex.pm/packages/markdown) is not precompiled and has not received updates in a while.
- [cmark](https://hex.pm/packages/cmark) is a fast CommonMark parser but it requires compiling the C library, is hard to extend, and was archived on Apr 2024.
## Comparison
|Feature|MDEx|Earmark|md|cmark|
| --- | --- | --- | --- | --- |
|Active|✅|⚠️¹|✅|❌|
|Pure Elixir|⚠️²|✅|✅|❌|
|Extensible|✅|✅|✅|❌|
|Syntax Highlighting|✅|❌|❌|❌|
|Code Block Decorators|✅|❌|❌|❌|
|Streaming (fragments)|✅|❌|❌|❌|
|Phoenix HEEx components|✅|❌|❌|❌|
|AST|✅|✅|✅|❌|
|AST to Markdown|✅|⚠️³|❌|❌|
|To HTML|✅|✅|✅|✅|
|To JSON|✅|❌|❌|❌|
|To XML|✅|❌|❌|✅|
|To Manpage|❌|❌|❌|✅|
|To LaTeX|❌|❌|❌|✅|
|To Quill Delta|✅|❌|❌|❌|
|To Slack|✅|❌|❌|❌|
|Emoji|✅|❌|❌|❌|
|GFM⁴|✅|✅|❌|❌|
|GLFM⁵|✅|❌|❌|❌|
|Discord⁶|⚠️⁷|❌|❌|❌|
1. Earmark is currently marked as retired
2. MDEx depends on [mdex_native](https://hex.pm/packages/mdex_native) which uses Rustler
3. Possible with [earmark_reversal](https://hex.pm/packages/earmark_reversal)
4. GitHub Flavored Markdown
5. GitLab Flavored Markdown
6. Discord Flavored Markdown
7. Partial support
## Benchmark
A [benchmark](benchmark.exs) is available to compare existing libs:
```text
Name ips average deviation median 99th %
cmark 8681.81 0.115 ms ±50.21% 0.111 ms 0.161 ms
mdex_native 6560.90 0.152 ms ±7.26% 0.149 ms 0.199 ms
mdex 395.26 2.53 ms ±10.58% 2.49 ms 2.98 ms
md 278.94 3.59 ms ±8.62% 3.55 ms 3.91 ms
earmark 69.86 14.32 ms ±6.02% 14.24 ms 15.41 ms
erlang-markdown 4.80 208.43 ms ±0.75% 208.19 ms 213.13 ms
Comparison:
cmark 8681.81
mdex_native 6560.90 - 1.32x slower +0.0372 ms
mdex 395.26 - 21.96x slower +2.41 ms
md 278.94 - 31.12x slower +3.47 ms
earmark 69.86 - 124.28x slower +14.20 ms
erlang-markdown 4.80 - 1809.56x slower +208.32 ms
Memory usage statistics:
Name average deviation median 99th %
cmark 0.00006 MB ±0.00% 0.00006 MB 0.00006 MB
mdex_native 0.00008 MB ±0.00% 0.00008 MB 0.00008 MB
mdex 2.36 MB ±0.00% 2.36 MB 2.36 MB
md 11.77 MB ±0.00% 11.77 MB 11.77 MB
earmark 7.91 MB ±0.00% 7.91 MB 7.91 MB
erlang-markdown 956.16 MB ±0.00% 956.16 MB 956.16 MB
Comparison:
cmark 0.00006 MB
mdex_native 0.00008 MB - 1.38x memory usage +0.00002 MB
mdex 2.36 MB - 38615.38x memory usage +2.36 MB
md 11.77 MB - 192838.89x memory usage +11.77 MB
earmark 7.91 MB - 129581.63x memory usage +7.91 MB
erlang-markdown 956.16 MB - 15665687.31x memory usage +956.16 MB
```
The most performance gain is using the `~MD` sigil to compile the Markdown instead of parsing it at runtime,
prefer using it when possible.
To finish, a friendly reminder that all libs have their own strengths and trade-offs so use the one that better suits your needs.
## Acknowledgements
- [comrak](https://crates.io/crates/comrak) crate for all the heavy work on parsing Markdown and rendering HTML
- [Floki](https://hex.pm/packages/floki) for the AST
- [Req](https://hex.pm/packages/req) for the pipeline API
- Logo based on [markdown-mark](https://github.com/dcurtis/markdown-mark)