MDEx

MDEx logo

Fast and Extensible Markdown for Elixir.

Hex Version Hex Docs MIT

Features

Foundation

The library is built on top of:

Installation

Add :mdex dependency:

def deps do

  [

    {:mdex, "~> 0.7"}

  ]

end

Usage

Mix.install([{:mdex, "~> 0.7"}])

iex> MDEx.to_html!("# Hello")

"<h1>Hello</h1>"

iex> MDEx.to_html!("# Hello :smile:", extension: [shortcodes: true])

"<h1>Hello πŸ˜„</h1>"

Plugins

Req-like Pipeline

MDEx.Pipe provides a high-level API to manipulate a Markdown document and build plugins that can be attached to a pipeline:

document = ~s"""

# Super Diagram



```mermaid

graph TD

  A[Enter Chart Definition] --> B(Preview)

  B --> C{decide}

  C --> D[Keep]

  C --> E[Edit Definition]

  E --> B

  D --> F[Save Image and Code]

  F --> B

```

"""



MDEx.new()

|> MDExMermaid.attach(mermaid_version: "11")

|> MDEx.to_html(document: document)

~MD Sigil

Convert and generate AST (MDEx.Document), Markdown (CommonMark), HTML, JSON, and XML formats.

iex> import MDEx.Sigil

iex> ~MD|# Hello from `~MD` sigil|

%MDEx.Document{

  nodes: [

    %MDEx.Heading{

      nodes: [

        %MDEx.Text{literal: "Hello from "},

        %MDEx.Code{num_backticks: 1, literal: "~MD"},

        %MDEx.Text{literal: " sigil"}

      ],

      level: 1,

      setext: false

    }

  ]

}

iex> import MDEx.Sigil

iex> ~MD|`~MD` also converts to HTML format|HTML

"<p><code>~MD</code> also converts to HTML format</p>"

iex> import MDEx.Sigil

iex> ~MD|and to XML as well|XML

"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE document SYSTEM \"CommonMark.dtd\">\n<document xmlns=\"http://commonmark.org/xml/1.0\">\n  <paragraph>\n    <text xml:space=\"preserve\">and to XML as well</text>\n  </paragraph>\n</document>"

~MD also accepts an assigns map to pass variables to the document:

iex> import MDEx.Sigil

iex> assigns = %{lang: "Elixir"}

iex> ~MD|Running <%= @lang %>|HTML

"<p>Running Elixir</p>"

See more info at https://hexdocs.pm/mdex/MDEx.Sigil.html

Code Block Decorators

Code block decorators allow you to customize the appearance and behavior of individual code blocks by adding special attributes to the info string (the part after the opening backticks).

Prerequisites

To use code block decorators, you must enable both :render options:

render: [

  github_pre_lang: true,

  full_info_string: true

]

Available Decorators

Decorator Description Supported Formatters Example
theme Override the syntax highlighting theme All theme=github_dark
pre_class Add custom CSS classes to <pre> element All pre_class="my-class"
highlight_lines Highlight single and/or range of lines All highlight_lines="1,3-5"
highlight_lines_style Custom inline styles for highlighted lines HTML inline only highlight_lines_style="background: yellow"
highlight_lines_class Custom CSS class for highlighted lines All highlight_lines_class="emphasis"
include_highlights Add syntax token names as data attributes All include_highlights

Examples

Following examples assume render: [github_pre_lang: true, full_info_string: true] is set.

Override Theme

Change the syntax highlighting theme for a specific code block:

```elixir theme=github_dark

def hello do

  "Hello, world!"

end

```

Output: <pre class="athl" style="color: #c9d1d9; background-color: #0d1117;">...

Add Custom CSS Classes

Add your own CSS classes to the <pre> element:

```javascript pre_class="code-example interactive"

console.log("Hello!");

```

Output: <pre class="athl code-example interactive">...

Highlight Specific Lines

Highlight individual lines or ranges (inclusive):

# Highlight lines 1, 4, 5, and 6



```python highlight_lines="1,4-6"

import math



def calculate(x):

    result = x * 2

    # return calculated result

    return math.sqrt(x)

```

With :html_inline formatter, lines get styles from the theme's highlight color, for eg:

<span style="background-color: #dae9f9;" data-line="1">...

With :html_linked formatter, the class highlighted is added to the highlighted lines, for eg:

<span class="line highlighted" data-line="1">...

Custom Highlight Styling

Use either highlight_lines_style or highlight_lines_class to customize the appearance of highlighted lines:

```ruby highlight_lines="2" highlight_lines_style="background: #ffeb3b; font-weight: bold;"

class User

  def initialize(name)

    @name = name

  end

end

```

Include Syntax Token Information

Add syntax token names in data-highlight attributes, useful for debugging or custom styling:

```rust include_highlights

let x: i32 = 42;

```

Output: <span data-highlight="keyword">let</span>

Combine Multiple Decorators

Use multiple decorators together:

```typescript theme=github_light pre_class="example" highlight_lines="2-3" include_highlights

interface User {

  name: string;    // highlighted

  email: string;   // highlighted

  age?: number;

}

```

Important Notes

  1. Formatter Support: Not all decorators work with all formatters:

    • highlight_lines_style only works with :html_inline formatter
    • theme only works with :html_inline formatter

  2. CSS Classes: The athl class is always added to <pre> elements when using syntax highlighting

  3. Line Numbers: The data-line attribute is added to each line for reference

  4. Order: It's expected the first word of the code fence info string to be the language name, followed by decorators.

    • For example, elixir theme=github_dark is valid, but theme=github_dark elixir is not.

  5. Performance: Decorators are processed at render time, so using many decorators may impact performance

Safety

For security reasons, every piece of raw HTML is omitted from the output by default:

iex> MDEx.to_html!("<h1>Hello</h1>")

"<!-- raw HTML omitted -->"

That's not very useful for most cases, but you have a few options:

Escape

The most basic is render raw HTML but escape it:

iex> MDEx.to_html!("<h1>Hello</h1>", render: [escape: true])

"&lt;h1&gt;Hello&lt;/h1&gt;"

Sanitize

But if the input is provided by external sources, it might be a good idea to sanitize it:

iex> MDEx.to_html!("<a href=https://elixir-lang.org>Elixir</a>", render: [unsafe: true], sanitize: MDEx.default_sanitize_options())

"<p><a href=\"https://elixir-lang.org\" rel=\"noopener noreferrer\">Elixir</a></p>"

Note that you must pass the unsafe: true option to first generate the raw HTML in order to sanitize it.

It does clean HTML with a conservative set of defaults that works for most cases, but you can overwrite those rules for further customization.

For example, let's modify the link rel attribute to add "nofollow" into the rel attribute:

iex> MDEx.to_html!("<a href=https://someexternallink.com>External</a>", render: [unsafe: true], sanitize: [link_rel: "nofollow noopener noreferrer"])

"<p><a href=\"https://someexternallink.com\" rel=\"nofollow noopener noreferrer\">External</a></p>"

In this case the default rule set is still applied but the link_rel rule is overwritten.

Unsafe

If those rules are too strict and you really trust the input, or you really need to render raw HTML, then you can just render it directly without escaping nor sanitizing:

iex> MDEx.to_html!("<script>alert('hello')</script>", render: [unsafe: true])

"<script>alert('hello')</script>"

Parsing

Converts Markdown to an AST data structure that can be inspected and manipulated to change the content of the document programmatically.

The data structure format is inspired on Floki (with :attributes_as_maps = true) so we can keep similar APIs and keep the same mental model when working with these documents, either Markdown or HTML, where each node is represented as a struct holding the node name as the struct name and its attributes and children, for eg:

%MDEx.Heading{

  level: 1

  nodes: [...],

}

The parent node that represents the root of the document is the MDEx.Document struct, where you can find more more information about the AST and what operations are available.

The complete list of nodes is listed in the documentation, section Document Nodes.

Formatting

Formatting is the process of converting from one format to another, for example from AST or Markdown to HTML. Formatting to XML and to Markdown is also supported.

You can use MDEx.parse_document/2 to generate an AST or any of the to_* functions to convert to Markdown (CommonMark), HTML, JSON, or XML.

Examples

GitHub Flavored Markdown with emojis

MDEx.to_html!(~S"""

# GitHub Flavored Markdown :rocket:



- [x] Task A

- [x] Task B

- [ ] Task C



| Feature | Status |

| ------- | ------ |

| Fast | :white_check_mark: |

| GFM  | :white_check_mark: |



Check out the spec at https://github.github.com/gfm/

""",

extension: [

  strikethrough: true,

  tagfilter: true,

  table: true,

  autolink: true,

  tasklist: true,

  footnotes: true,

  shortcodes: true,

],

parse: [

  smart: true,

  relaxed_tasklist_matching: true,

  relaxed_autolinks: true

],

render: [

  github_pre_lang: true,

  unsafe: true,

]) |> IO.puts()

"""

<p>GitHub Flavored Markdown πŸš€</p>

<ul>

  <li><input type="checkbox" checked="" disabled="" /> Task A</li>

  <li><input type="checkbox" checked="" disabled="" /> Task B</li>

  <li><input type="checkbox" disabled="" /> Task C</li>

</ul>

<table>

  <thead>

    <tr>

      <th>Feature</th>

      <th>Status</th>

    </tr>

  </thead>

  <tbody>

    <tr>

      <td>Fast</td>

      <td>βœ…</td>

    </tr>

    <tr>

      <td>GFM</td>

      <td>βœ…</td>

    </tr>

  </tbody>

</table>

<p>Check out the spec at <a href="https://github.github.com/gfm/">https://github.github.com/gfm/</a></p>

"""

Code Syntax Highlighting

MDEx.to_html!(~S"""

```elixir

String.upcase("elixir")

```

""",

syntax_highlight: [

  formatter: {:html_inline, theme: "catppuccin_latte"}

]

) |> IO.puts()

"""

<pre class=\"autumn highlight\" style=\"background-color: #282C34; color: #ABB2BF;\">

  <code class=\"language-elixir\" translate=\"no\">

    <span class=\"namespace\" style=\"color: #61AFEF;\">String</span><span class=\"operator\" style=\"color: #C678DD;\">.</span><span class=\"function\" style=\"color: #61AFEF;\">upcase</span><span class=\"\" style=\"color: #ABB2BF;\">(</span><span class=\"string\" style=\"color: #98C379;\">&quot;elixir&quot;</span><span class=\"\" style=\"color: #ABB2BF;\">)</span>

  </code>

</pre>

"""

Pre-compilation

Pre-compiled binaries are available for the following targets, so you don't need to have Rust installed to compile and use this library:

Note: The pre-compiled binaries for Linux are compiled using Ubuntu 22 on libc 2.35, which requires minimum Ubuntu 22, Debian Bookworm or a system with a compatible libc version. For older Linux systems, you'll need to compile manually.

Compile manually

But in case you need or want to compile it yourself, you can do the following:

  1. Install Rust

  2. Install a C compiler or build packages

It depends on your OS, for example in Ubuntu you can install the build-essential package.

  1. Run:
export MDEX_BUILD=1

mix deps.get

mix compile

Legacy CPUs

Modern CPU features are enabled by default but if your environment has an older CPU, you can use legacy artifacts by adding the following configuration to your config.exs:

config :mdex, use_legacy_artifacts: true

Used By

Are you using MDEx and want to list your project here? Please send a PR!

Motivation

MDEx was born out of the necessity of parsing CommonMark files, to parse hundreds of files quickly, and to be easily extensible by consumers of the library.

Comparison

Feature MDEx Earmark md cmark
Active βœ… βœ… βœ… ❌
Pure Elixir ❌ βœ… βœ… ❌
Extensible βœ… βœ… βœ… ❌
Syntax Highlighting βœ… ❌ ❌ ❌
Code Block Decorators βœ… ❌ ❌ ❌
AST βœ… βœ… βœ… ❌
AST to Markdown βœ… ⚠️² ❌ ❌
To HTML βœ… βœ… βœ… βœ…
To JSON βœ… ❌ ❌ ❌
To XML βœ… ❌ ❌ βœ…
To Manpage ❌ ❌ ❌ βœ…
To LaTeX ❌ ❌ ❌ βœ…
Emoji βœ… ❌ ❌ ❌
GFMΒ³ βœ… βœ… ❌ ❌
GitLab⁴ ⚠️¹ ❌ ❌ ❌
Discord⁡ ⚠️¹ ❌ ❌ ❌
  1. Partial support
  2. Possible with earmark_reversal
  3. GitHub Flavored Markdown
  4. GitLab Flavored Markdown
  5. Discord Flavored Markdown

Benchmark

A simple script is available to compare existing libs:

Name              ips        average  deviation         median         99th %

cmark          7.17 K       0.139 ms     Β±4.20%       0.138 ms       0.165 ms

mdex           2.71 K        0.37 ms     Β±7.95%        0.36 ms        0.45 ms

md            0.196 K        5.11 ms     Β±2.51%        5.08 ms        5.55 ms

earmark      0.0372 K       26.91 ms     Β±2.09%       26.77 ms       30.25 ms



Comparison:

cmark          7.17 K

mdex           2.71 K - 2.65x slower +0.23 ms

md            0.196 K - 36.69x slower +4.98 ms

earmark      0.0372 K - 193.04x slower +26.77 ms

The most performance gain is using the ~MD sigil to compile the Markdown instead of parsing it at runtime, prefer using it when possible:

Comparison:

mdex_sigil_MD    176948.46 K

cmark                31.47 K - 5622.76x slower +31.77 ΞΌs

mdex_to_html/1        7.32 K - 24184.36x slower +136.67 ΞΌs

md                    2.05 K - 86176.93x slower +487.01 ΞΌs

earmark               0.21 K - 855844.67x slower +4836.68 ΞΌs

To finish, a friendly reminder that all libs have their own strengths and trade-offs so use the one that better suits your needs.

Acknowledgements