WikitextEx

Hex.pmDocumentation

A MediaWiki wikitext parser for Elixir that converts wikitext markup into structured AST nodes. WikitextEx supports templates, links, formatting, tables, and other wikitext elements commonly found in MediaWiki content.

⚠️ Current Status: Beta - WikitextEx successfully parses many common wikitext patterns but has known limitations with certain edge cases. See Known Limitations for details.

Features

Installation

Add wikitext_ex to your list of dependencies in mix.exs:

def deps do
  [
    {:wikitext_ex, "~> 0.1.0"}
  ]
end

Usage

Basic Parsing

# Parse simple wikitext
{:ok, ast, _, _, _, _} = WikitextEx.Parser.parse("'''Bold text''' and ''italic text''")

# The result is a list of AST nodes
[
  %WikitextEx.AST{type: :bold, children: [%WikitextEx.AST{type: :text, value: %WikitextEx.AST.Text{content: "Bold text"}}]},
  %WikitextEx.AST{type: :text, value: %WikitextEx.AST.Text{content: " and "}},
  %WikitextEx.AST{type: :italic, children: [%WikitextEx.AST{type: :text, value: %WikitextEx.AST.Text{content: "italic text"}}]}
]

Templates

# Parse templates with arguments
{:ok, ast, _, _, _, _} = WikitextEx.Parser.parse("{{template|arg1|key=value}}")

# Results in template AST node
%WikitextEx.AST{
  type: :template,
  value: %WikitextEx.AST.Template{
    name: "template",
    args: [
      {:positional, "arg1"},
      {:named, %{"key" => "value"}}
    ]
  }
}

Links and Categories

# Parse various link types
{:ok, ast, _, _, _, _} = WikitextEx.Parser.parse("[[Article]] [[Category:Example]] [[File:image.jpg|thumb]]")

# Results in different AST node types:
# - :link for regular internal links
# - :category for category links
# - :file for file/media links

Headers

# Parse headers
{:ok, ast, _, _, _, _} = WikitextEx.Parser.parse("== Section Header ==")

%WikitextEx.AST{
  type: :header,
  value: %WikitextEx.AST.Header{level: 2},
  children: [%WikitextEx.AST{type: :text, value: %WikitextEx.AST.Text{content: "Section Header"}}]
}

Tables

wikitext = """
{|
! Header 1 !! Header 2
|-
| Cell 1 || Cell 2
|}
"""

{:ok, ast, _, _, _, _} = WikitextEx.Parser.parse(wikitext)
# Results in table AST with rows and cells

AST Structure

WikitextEx produces a structured AST where each node follows this pattern:

%WikitextEx.AST{
  type: atom(),           # The type of element (:text, :template, :link, etc.)
  value: struct() | nil,  # Type-specific data (e.g., %AST.Text{content: "..."})
  children: [%AST{}]      # Nested AST nodes
}

Supported AST Node Types

Advanced Usage

Working with AST

# Extract text content from headers or other containers
WikitextEx.AST.text_content(ast_node.children)

# Navigate the AST tree
defmodule WikitextWalker do
  def find_templates(ast_nodes) do
    Enum.flat_map(ast_nodes, fn
      %WikitextEx.AST{type: :template} = node -> [node]
      %WikitextEx.AST{children: children} -> find_templates(children)
      _ -> []
    end)
  end
end

Custom Processing

defmodule WikitextProcessor do
  def extract_links(wikitext) do
    case WikitextEx.Parser.parse(wikitext) do
      {:ok, ast, _, _, _, _} ->
        ast
        |> find_links()
        |> Enum.map(& &1.value.target)

      {:error, _} ->
        []
    end
  end

  defp find_links(ast_nodes) do
    Enum.flat_map(ast_nodes, fn
      %WikitextEx.AST{type: :link} = node -> [node]
      %WikitextEx.AST{children: children} -> find_links(children)
      _ -> []
    end)
  end
end

Development

# Clone the repository
git clone https://github.com/your-username/wikitext_ex.git
cd wikitext_ex

# Install dependencies
mix deps.get

# Run tests
mix test

# Generate documentation
mix docs

Known Limitations

WikitextEx works well for typical wiki content, but has some known limitations:

Parser Edge Cases

Parsing Behavior

Recommendations

Testing

WikitextEx includes a comprehensive test suite with 58 tests covering various wikitext patterns and edge cases. Run the tests with:

mix test

Contributing

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Write tests for your changes
  4. Ensure all tests pass (mix test)
  5. Commit your changes (git commit -am 'Add amazing feature')
  6. Push to the branch (git push origin feature/amazing-feature)
  7. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments