ldoc

This source builds ldoc 0.1.0, which collects API documentation for Erlang, modern LFE, and classic LFE from trusted local BEAM artifacts. It assembles native snapshots and renders a static HTML reference. collect/1 prefers embedded EEP-48 Docs, then an associated package sidecar, then classic LFE LDoc from the same BEAM. It falls back only when a carrier is absent. read-docs/1 reads embedded Docs only. Neither reader loads the target module or searches the code path.

Install from a published release

The normal user route clones the latest published GitHub release and installs the standalone command under an absolute prefix. It needs Git, Make, Erlang/OTP with its compiler and crypto application, and Rebar3. Rebar3 3.28.0 is the version verified here; this source is tested with OTP 27, 28, and 29. For Erlang source-comment conversion, the OTP EDoc application must provide edoc_doclet_chunks and edoc_layout_chunks; ldoc reports an error when those chunk tools are unavailable. Automatic latest-release selection also needs gh authenticated to GitHub. To skip gh, set ldoc_tag in the calling shell to an actual published tag shown on the ldoc Releases page.

(
  set -eu
  ldoc_prefix="${PREFIX:-$HOME/.local}"
  case "$ldoc_prefix" in
    /*) ;;
    *) ldoc_prefix="$PWD/$ldoc_prefix" ;;
  esac
  ldoc_work="$(mktemp -d /tmp/ldoc-install.XXXXXXXX)"
  trap 'rm -rf -- "$ldoc_work"' EXIT
  ldoc_tag="${ldoc_tag:-$(gh release view --repo lfex/ldoc --json tagName --jq '.tagName')}"
  git clone --depth 1 --branch "$ldoc_tag" https://github.com/lfex/ldoc.git "$ldoc_work/source"
  make -C "$ldoc_work/source" install PREFIX="$ldoc_prefix"
)

Use this route only with a tag for an actual published ldoc release. If no release is published yet, or you are working from a checkout, use the source checkout route below. The Releases page is the source for choosing a real tag; do not substitute a development branch or commit.

The recipe resolves a relative PREFIX to an absolute path before make -C changes directories, then removes only its unique temporary clone/build root. The installed command remains outside that root. After installation succeeds, set PATH and check the command using the matching prefix. For the default $HOME/.local prefix:

export PATH="$HOME/.local/bin:$PATH"
ldoc --help

For a custom prefix, replace the example with the normalized absolute prefix used for installation; do not use a relative PATH entry that may point somewhere else after a later directory change:

export PATH="/absolute/prefix/bin:$PATH"
ldoc --help

Install from a source checkout

For development, install the current checkout directly. make check is a separate optional compile-and-test gate, not a prerequisite hidden in the install recipe.

make install PREFIX="$HOME/.local"
"$HOME/.local/bin/ldoc" --help

Maintainer package and docs checks

From a source checkout, the maintained reference and combined Hex provider flow are:

make check
make docs
make publish PUBLISH_FLAGS="--dry-run --yes"

make docs builds a fresh versioned reference under _build/ldoc/docs.XXXXXXXX/reference/site and leaves it available for browsing. Set DOCS_PARENT to choose another persistent parent, or DOCGEN to select a built ldoc executable. The release version comes from the compiled app resource. make publish builds a separate fresh reference under a unique publish.* staging directory and calls the combined source-and-docs Hex publish provider once. Its EXIT trap removes only that staging directory; persistent make docs outputs remain. --dry-run --yes builds both provider payloads without uploading either package. A real publish requires the maintainer's normal Hex authorization.

To inspect the official source package separately, run rebar3 hex build package. The default output is _build/default/lib/ldoc/hex/ldoc-0.1.0.tar after the application is built. The package contains the application source, tests, README, overview, LICENSE, Makefile, and Rebar configuration/lock. The companion docs package is generated from the same versioned site used by the combined provider. These source/docs artifacts describe the release; ldoc does not provide a prebuilt executable through this workflow.

Build a local reference

From a trusted flat single-application checkout, run:

"$HOME/.local/bin/ldoc" build "$PWD" \
  --output "$PWD/ldoc-1.2.0" \
  --overview "$PWD/README.md" \
  --source-root "$PWD" \
  --source-base "https://example.test/my-app/"

The output root must be new, and its parent directory must already exist. Open ldoc-1.2.0/site/index.html. The 1.2.0 part is a caller-chosen output directory label; the exact release version in the reference comes from the compiled application resource. Use another fresh output directory for another run; ldoc never replaces an existing path. The root retains staged package BEAMs and the readable native snapshot used for the site. Without --overview, ldoc uses the project's README.md when present. The source root and source base are explicit caller declarations; ldoc does not infer or verify a repository host or revision.

The command owns the ordinary rebar3 compile step and accepts one root src/*.app.src plus nonempty, direct src/*.erl and src/*.lfe module files under the default _build/default/lib/APP/ebin layout. Umbrella, nested, custom-layout, and non-default-profile builds are not supported by this command. It verifies that source modules, the compiled application module list, and compiled BEAM files agree before claiming the output path.

Embedded Docs chunks remain authoritative. For an Erlang module without one, the command generates an EDoc chunk in its owned output package from matching source and available compile-time include and macro context. collect/2 records a caller-declared EDoc conversion context only when that sidecar is selected; it does not run EDoc or authenticate source paths. EDoc generation failures retain the newly owned partial output for diagnosis; unsupported preprocessing contexts and unavailable EDoc chunk tools fail visibly.

ldoc build PROJECT --output FRESH_ROOT --overview FILE --namespace TEXT --package TEXT --source-root ROOT --source-base URL lists the optional arguments. Namespace defaults to local; package and exact version default to the compiled application identity. The installed command path, install prefix, and generated output paths support spaces and Unicode when native filename encoding is UTF-8. A project path used for a cold dependency build is subject to the pinned LFE 2.2.2 dependency's app_compile hook, which constructs a command with an unquoted path; use a space-free source/cache/build root for that cold build.

'ldoc-beam':'read-docs'("_build/default/lib/my_app/ebin/my_module.beam").
'ldoc-beam':collect("_build/default/lib/my_app/ebin/my_module.beam").

Store exact release snapshots

Assemble the complete native docset, then save and retrieve it by the exact release key. The store writes a readable, versioned LFE data form under a bounded SHA-256 filename; it does not infer aliases or order versions.

Release = #{namespace => <<"local">>, package => <<"my_app">>,
            version => <<"1.2.0">>},
Inputs = ["_build/default/lib/my_app/ebin/my_module.beam"],
{ok, Docset} = 'ldoc-docset':assemble(Release, Inputs),
ok = 'ldoc-store':put("/var/tmp/ldoc-snapshots", Docset),
{ok, Docset} = 'ldoc-store':get("/var/tmp/ldoc-snapshots", Release).

An identical retry returns ok without rewriting the snapshot. A different docset under the same exact release key returns {error, {release_conflict, Release}}, leaving the existing release data unchanged. Different version keys coexist and remain independently retrievable. Snapshots retain their complete producer maps as native LFE terms, including metadata, provenance and structured documentation.

The caller chooses a trusted local store directory on a filesystem that supports exclusive file creation and same-directory hard links. The hard-link insertion is the immutable publication point. This protects concurrent writers and process interruption on a live filesystem; power-loss durability depends on additional guarantees from the filesystem and operating system.

The readable data subset follows LFE's stock literal writers: comments and whitespace, direct literal binary strings, byte values, and an optional final partial-byte size segment are supported. Reader-eval forms (#.) and computed binary expressions are invalid snapshots. Maps containing the LFE struct marker key __struct__ are retained as ordinary data using LFE's plain readable printer because the pretty-printer treats that key as struct syntax.

Build a native analysis view

Retrieve the exact release, then project it through ldoc-analysis:view/1. The view groups languages, modules and candidate definitions while retaining the binary kind/name/arity identity and selected documentation as native terms. Overloads remain separate, and a classic macro keeps unknown arity.

{ok, Docset} = 'ldoc-store':get(StoreRoot, Release),
{ok, View} = 'ldoc-analysis':view(Docset).

The projection is a selected presentation boundary. It copies module name, format and documentation; definition kind, name, arity, signatures, documentation and producer patterns when present; plus optional relative source references. It excludes internal entries and suppresses definitions for hidden modules. Candidate entries with hidden documentation remain visible with that state. Structured EDoc bodies, signatures and classic patterns stay in native form. Opaque metadata, raw spec terms, annotations, carrier paths and arbitrary source data stay in the stored docset.

Source references are absent by default. To include them, supply a declared absolute source root with view/2:

{ok, ViewWithSources} =
    'ldoc-analysis':view(Docset, #{source_root => SourceRoot}).

Only recorded source-file provenance beneath that root can produce a relative path; a known positive line is included when available. This is lexical projection: it does not inspect the filesystem, resolve symlinks, verify file existence, or authenticate the caller's declaration. It never infers a source from a BEAM or sidecar path, and it leaves path-like prose unchanged. A rendered source link requires both a selected relative source record and the caller's explicit source_base; the link does not verify a host, revision, or checkout.

An LFE consumer can traverse the native view without reading collector or store internals:

DefinitionIndex = maps:from_list([
    {{Language, maps:get(name, Module), maps:get(kind, Definition),
      maps:get(name, Definition), maps:get(arity, Definition)}, Definition}
    || #{name := Language, modules := Modules} <- maps:get(languages, View),
       Module <- Modules,
       Definition <- maps:get(definitions, Module)]).

The store retains the complete producer record, including raw specs and annotations. This view exposes the producer's existing signature binaries; typed-spec presentation and JSON or other wire formats need a later, consumer-specific contract.

Render a local HTML reference

ldoc-html:render/1 and render/2 accept the selected native view from ldoc-analysis:view/1 and return {ok, Files}. Files is a sorted list of {RelativeFilenameBinary, UTF8BytesBinary} entries: index.html, one m-<SHA-256>.html module page per module, ldoc.css, and ldoc.js. Invalid option maps return {error, invalid_options}; malformed or unsupported views return {error, invalid_view}. Rendering is in-memory and does not read files, load target modules, access the network, or mutate the stored release.

Options are a map containing only:

The requested locale still selects documentation by exact key, then und, then the first key in lexical order. A selected body receives an HTML lang attribute only when the key matches ldoc's conservative subset: a two- or three-letter primary language, optionally followed by a four-letter script and a two-letter or three-digit region. The primary und is never annotated. Other valid UTF-8 keys still select and render their text without a lang attribute; this subset is not full BCP 47 validation. The generated interface chrome is English and left-to-right. Preserving Unicode and selecting a locale do not apply an automatic paragraph-level bidirectional-text policy.

Example: export the returned bundle beneath a caller-selected directory and open the generated overview. The example's source URL is an explicit caller declaration, not a value inferred from the store or artifact paths.

{ok, Docset} = 'ldoc-store':get(StoreRoot, Release),
{ok, View} = 'ldoc-analysis':view(Docset),
{ok, Files} = 'ldoc-html':render(View,
                                  #{locale => <<"en">>,
                                    overview => <<"# Local API\n\nBrowse the modules below.">>,
                                    source_base => <<"https://example.test/project/">>}),
OutputDir = "/tmp/my_app-reference",
ok = file:make_dir(OutputDir),
ok = lists:foreach(
       fun({Name, Bytes}) ->
           ok = file:write_file(filename:join(OutputDir, binary_to_list(Name)), Bytes)
       end,
       Files).

Open file:///tmp/my_app-reference/index.html in a browser. For a local HTTP server, run python3 -m http.server 8000 --directory /tmp/my_app-reference from another terminal and visit http://localhost:8000/.

The Markdown renderer implements a safe documented subset: paragraphs, ATX headings, unordered and decimal ordered lists, blockquotes, fenced code, pipe tables, inline code, strong/emphasis, and inline links. It is not a full CommonMark implementation. Raw HTML is shown as escaped text, unsupported syntax remains text, and images do not fetch remote content. Links are retained only for validated absolute HTTP, HTTPS, or mailto targets; unsafe and unsupported targets remain text. application/erlang+html documentation uses an allowlisted walker for nested native EDoc tags and validated link attributes; unknown tags unwrap safe children, while unsupported nodes are escaped. Producer HTML attributes such as id, class, style, and event handlers are not copied into the generated page.

A source action appears only when both the accepted view contains a valid relative source reference and the caller supplies source_base. Each path component is percent-encoded using unreserved ASCII, slash separators are preserved, and a known positive line becomes #L<n>. No source host, revision, or target is inferred or verified. This is link construction from declared inputs, not source checkout authentication. Path-like prose and code are deliberately retained when they are part of selected documentation; opaque producer metadata is excluded from the selected HTML view. The renderer does not promise to scrub or redact documentation text. Raw typed specs and annotations remain in storage and are not presented as inferred types by this API; signatures and classic patterns are the selected native view's documented presentation fields.

Build a versioned local site

ldoc-site:build/3 composes exact stored releases through the public store, analysis-view, and HTML-renderer APIs. Supply a nonempty proper list of release specifications. Each map has a required exact release key and may include overview, source_root, and source_base. Release keys are not normalized or semver-sorted; the catalogue groups exact namespace/package coordinates and lists their exact versions bytewise. The optional fourth argument accepts only locale. Every release is prepared before the builder claims or changes the output directory.

From the ldoc checkout, start an Erlang shell with rebar3 shell; the compiled application is then available on the code path. Paste the following block into that shell:

{ok, _Started} = application:ensure_all_started(ldoc),
Release1 = #{namespace => <<"local">>, package => <<"site-demo">>,
             version => <<"1.0">>},
Release2 = Release1#{version := <<"2.0">>},
%% This runnable demonstration uses one installed ldoc BEAM for both keys.
%% Real releases should each be assembled from their matching compiled inputs.
Inputs = [code:which('ldoc-docset')],
{ok, Docset1} = 'ldoc-docset':assemble(Release1, Inputs),
{ok, Docset2} = 'ldoc-docset':assemble(Release2, Inputs),
Work = filename:join("/tmp", "ldoc-site-demo-" ++
                     integer_to_list(erlang:system_time(microsecond)) ++ "-" ++
                     integer_to_list(erlang:unique_integer([positive]))),
ok = file:make_dir(Work),
StoreRoot = filename:join(Work, "snapshots"),
ok = file:make_dir(StoreRoot),
ok = 'ldoc-store':put(StoreRoot, Docset1),
ok = 'ldoc-store':put(StoreRoot, Docset2),
Target = filename:join(Work, "site"),
Specs = [#{release => Release1,
           overview => <<"# My application\n\nBrowse the exact API release.">>},
         #{release => Release2}],
{ok, IndexPath} = 'ldoc-site':build(StoreRoot, Specs, Target),
io:format("Open this local index in a browser: ~ts~n", [IndexPath]).

The block creates a fresh work directory and snapshots child under /tmp, stores both exact demo releases, and leaves the new site target for the builder to claim. Open the printed absolute IndexPath in a browser as a local file. For HTTP, use the printed index's containing directory in another terminal, for example python3 -m http.server 8000 --directory /tmp/ldoc-site-demo-<id>/site, then visit http://localhost:8000/. The two demo keys intentionally share the same installed BEAM; for real releases, pass each release's own matching compiled inputs to ldoc-docset:assemble/2. The catalogue has direct keyboard- and no-script-friendly links to each exact release. Within a release, each page's visible version link returns to the catalogue. source_root and source_base remain caller declarations; a source link appears only for a relative reference selected by the public analysis view beneath that root. The builder does not infer a host, revision, or source target.

The target must not already exist, and its parent must already be a directory. The caller provides a stable trusted parent; the builder does not resolve symlinked ancestors or defend a parent another process can replace while the build runs. A build that loses an exclusive target claim returns {error, output_exists} without changing the winner. Other validation errors are returned in this order: invalid_options, invalid_store_root, invalid_target, invalid_releases, then {duplicate_release, Release}. After validation, a read/render error is tagged with the exact release and stage. Parent, directory creation, open, write, and close errors identify the operation and path. A failed owned build removes its new tree where possible; cleanup failure is reported with the original operation error. Unexpected exceptions are re-raised after closing the current descriptor and attempting owned-tree cleanup.

The generated site is for local browsing after a successful return. It is not an atomic multi-file publication or a power-loss durability guarantee. An interrupted process may leave an incomplete owned target for its caller to inspect and remove. Only generated release routes use bounded SHA-256 identities; no release label becomes a filesystem path. The previous single-release ldoc-html:render/1,2 API remains in-memory and keeps its original header bytes unless its explicit version_index option is true.

The independently authored layout takes measured page organization cues from cljdoc's Reitit 0.11.0 reference and the inspected upstream source at 74d0cec6bd252321375b68ee3facb8ae8faf3296 (EPL-2.0). ldoc uses no copied upstream source or assets.

Generate an EEP-48 sidecar with EDoc

For a source-comment module, compile and run OTP EDoc as a caller step. Use a fresh trusted package or output directory so an old chunk cannot be mistaken for the current conversion. The package layout is ebin/<module>.beam beside doc/chunks/<module>.chunk.

{ok, Module, Beam} =
    compile:file(SourcePath, [binary, {i, IncludeDir}]),
BeamPath = filename:join(PackageEbinDir, atom_to_list(Module) ++ ".beam"),
ok = filelib:ensure_dir(BeamPath),
ok = file:write_file(BeamPath, Beam),
ok = filelib:ensure_dir(filename:join(PackageDocDir, "chunks/placeholder")),
ok = edoc:files([SourcePath],
                [{dir, PackageDocDir},
                 {doclet, edoc_doclet_chunks},
                 {layout, edoc_layout_chunks},
                 {preprocess, true},
                 {includes, [IncludeDir]}]).

Then collect the package BEAM. The second form records the caller's source and include declaration with the sidecar result; it does not run EDoc or verify those paths.

'ldoc-beam':collect(BeamPath).
'ldoc-beam':collect(BeamPath, {edoc, SourcePath, [IncludeDir]}).

The declaration is preserved as conversion => #{tool => edoc, source_path => SourcePath, include_paths => IncludeDirs, evidence => caller_declared} only when a sidecar is selected. Embedded results keep their original source map. A nonempty string or binary source path and a proper list of nonempty string or binary include paths are required; the include list may be empty.

Collect classic LFE documentation

collect/1 also reads the versioned LDoc chunk emitted by LFE 1.2.0, after embedded EEP-48 Docs and an eligible same-package sidecar are absent. The legacy record is decoded from bytes read from the supplied BEAM; the module is not loaded. read-docs/1 remains an embedded Docs-only operation and returns missing_docs for a classic-only artifact.

The result uses the shared module and entry maps. Its source is #{carrier => embedded_beam, path => Path, chunk => <<"LDoc">>, legacy_version => lfe_docs_v1, format_policy => lodox_markdown}. The producer did not record a module annotation or a documentation locale, so the module annotation is undefined; nonempty line lists use #{<<"und">> => Text}. Module and entry line lists are joined with newlines. Only an empty line list becomes none; a nonempty list such as [<<>>] remains an und entry with an empty binary body. The exact original lines remain in metadata.legacy_doc_lines. Entry patterns retain the original function and macro terms, including guarded clauses. Entry metadata retains legacy_doc_lines and legacy_exported; function lines become annotations and signatures are empty. Legacy macro arity is unknown, even if a clause looks fixed-arity. Empty producer doc lists remain none. Lodox interpreted these docstrings as Markdown.

The selection order is embedded Docs, an associated sidecar under the immediate package root, then LDoc from the same BEAM bytes. Only an exact missing_docs result from sidecar selection advances to LDoc; present invalid Docs, sidecar read failures, and rejected sidecar associations stay errors. A missing LDoc chunk returns missing_docs; a malformed or unsupported versioned payload returns invalid_ldoc. The LDoc adapter preserves producer patterns as data and does not execute them.

Reproduce the classic fixture

The shipped test/fixtures/ldoc-legacy.beam.fixture is a genuine compiler output for test/fixtures/ldoc-legacy.lfe.fixture. Regeneration needs the pinned local LFE repository and an OTP 19.3 installation. Set LFE_REPO to a clone containing commit a4b20a050d38e3f494cdf4a1a99adbe11e7b6ced and OTP19_BIN to the OTP 19.3 bin directory:

LFE_REPO=/path/to/lfe
OTP19_BIN=/path/to/otp-19.3/bin
WORK=$(mktemp -d)
mkdir -p "$WORK/lfe" "$WORK/source"
git -C "$LFE_REPO" archive --format=tar a4b20a050d38e3f494cdf4a1a99adbe11e7b6ced | tar -xf - -C "$WORK/lfe"
cp test/fixtures/ldoc-legacy.lfe.fixture "$WORK/source/ldoc-legacy.lfe"
(cd "$WORK/lfe" && PATH="$OTP19_BIN:$PATH" ERL_FLAGS='+S 2:2' make)
(cd "$WORK/source" && PATH="$OTP19_BIN:$PATH" ERL_FLAGS='+S 2:2' "$OTP19_BIN/erl" -noshell -pa "$WORK/lfe/ebin" -eval '
    Source = "ldoc-legacy.lfe",
    {ok, [{ok, Module, Beam}]} = lfe_comp:file(Source, [binary, report]),
    true = (atom_to_list(Module) =:= "ldoc-legacy"),
    ok = file:write_file("ldoc-legacy.beam", Beam),
    {ok, {Module, [{"LDoc", Chunk}]}} = beam_lib:chunks(Beam, ["LDoc"]),
    ok = file:write_file("ldoc-legacy.ldoc", Chunk),
    halt().')
shasum -a 256 "$WORK/source/ldoc-legacy.ldoc"

The expected raw LDoc chunk is 555 bytes with SHA-256 a4191d04427134e4ffb4e28011f1b97dcff754609b394e6e23e801c1fb2a8ea6. Compare those extracted bytes with a separately extracted chunk from the shipped BEAM fixture. The full BEAM may differ because compiler metadata records the source path.

For sidecars, ldoc uses the module identity read from the supplied BEAM and checks only the immediate package root above its ebin directory. It does not infer identity from the filename, walk ancestors, or consult the VM code path. Embedded docs are authoritative, including hidden and none; only a missing embedded chunk advances to the sidecar. A malformed embedded chunk remains invalid_docs even if a sidecar exists.

Filesystem errors retain tagged reasons (read_beam or read_sidecar). Invalid BEAMs, missing docs, invalid docs, invalid package associations, and invalid EDoc provenance have distinct tagged errors. Sidecars contain no module/build identity to authenticate their contents, so collection assumes a trusted package with stable contents. Do not use this reader for hostile or public artifacts; atom creation and resource bounds need a separate isolation design.