Obscura

Obscura is a library-first PII detection and anonymization toolkit for Elixir.

The supported 0.1.x surface is explicitly frozen. See docs/public-api-stability.md for stable modules, experimental adapters, option schemas, struct guarantees, and the deprecation policy.

Obscura provides a dependency-light string API for common pattern-based entities:

• email
• phone
• credit card
• US SSN
• IBAN
• IP address
• URL
• domain

Product Profiles

Obscura has three stable profiles:

Obscura.analyze("Contact jane@example.com", profile: :fast)

Model profiles require explicit reusable runtime preparation. Ordinary analyze and redact calls never download models:

{:ok, runtime} =
  Obscura.Profile.prepare(:balanced,
    allow_download: true,
    real_model_backend: :emily,
    emily_fallback: :raise,
    compile: [batch_size: 1, sequence_length: 128]
  )

Obscura.analyze("Rachel works at Google in Paris.", profile: runtime)

Downloading third-party assets is never inferred from selecting a profile. The first online preparation must pass allow_download: true; the default is cache-only. These are large assets: the measured Apple/Emily cache used about 1.4 GB for :balanced and 2.84 GB of active files for :accurate. Network and storage performance determine cold preparation time, so applications should prepare during deployment or supervised startup, not while handling a request. For visible stages, observed download bytes, model index, cache location, timeouts, and a machine-readable mode, use:

mix obscura.profile.prepare \
  --profile balanced \
  --backend emily \
  --allow-download \
  --timeout 1800000 \
  --inactivity-timeout 300000

Use --offline in deployments with pre-provisioned caches. Interrupted unreferenced cache files are quarantined on retry. Online preparation retries one transient model or tokenizer load failure; offline preparation never retries asset access. Quarantined files remain on disk for diagnosis until an operator removes them. Prepare once and reuse the returned runtime; never prepare inside a request. Setup, cache, recovery, and backend requirements are documented in docs/optional-dependencies-and-assets.md.

Applications which must continue starting while models load can supervise Obscura.Profile.Preparer and call await/2 before accepting model-backed work:

children = [
  {Obscura.Profile.Preparer,
   name: MyApp.ObscuraRuntime,
   profile: :balanced,
   prepare_options: [allow_download: true, real_model_backend: :emily]}
]

{:ok, runtime} = Obscura.Profile.Preparer.await(MyApp.ObscuraRuntime, :timer.minutes(30))

Check readiness without analyzing PII:

mix obscura.profile.check --profile fast
mix obscura.profile.check --profile balanced --backend emily --json

See docs/profiles.md, docs/benchmark-status.md, and docs/optional-dependencies-and-assets.md before selecting a model profile. Current accuracy, startup, concurrency, p99, memory, recovery, sustained-load, and capacity conclusions are summarized in docs/benchmark-status.md. Runtime failures are covered by docs/runtime-diagnostics.md; residual risks and deployment limits are in docs/known-limitations.md.

:balanced and :accurate are stable, technically production-oriented profile contracts. :accurate has the highest measured general accuracy, while :balanced remains the practical model-backed recommendation because it uses one model and has lower latency. Their optional model assets are third-party: Obscura neither bundles nor licenses them, and the TNER checkpoint license is not established. Deployers must determine whether their selected assets and use are permitted. See docs/model-asset-licensing.md.

:accurate conditionally invokes its location specialist only when the primary model misses location. It beats :balanced F1 on all three authoritative datasets, but the gain is small on Synth/Nemotron and the second model increases cost. See docs/benchmark-status.md.

Additional experimental model adapters and benchmark profiles remain available for research and controlled evaluation. They are outside the stable compatibility promise. See docs/profiles.md and docs/model-backed-recognition.md.

Installation

Obscura is an early release. Add it to an Elixir project with:

def deps do
  [
    {:obscura, "~> 0.1.0"}
  ]
end

Analyze

{:ok, results} = Obscura.analyze("Contact jane@example.com", entities: [:email])

[%Obscura.Analyzer.Result{entity: :email, start: 8, end: 24}] = results

Offsets are byte offsets. This is intentional because Elixir binaries are byte indexed and anonymization uses binary slicing.

Anonymize

text = "Contact jane@example.com"
{:ok, detections} = Obscura.analyze(text, entities: [:email])

{:ok, result} =
  Obscura.anonymize(text, detections,
    operators: %{email: %{type: :replace, value: "[EMAIL]"}}
  )

result.text
#=> "Contact [EMAIL]"

Supported operators are:

• :replace - %{type: :replace, value: "[REDACTED]"}
• :redact - %{type: :redact}
• :mask - %{type: :mask, char: "*", keep_last: 4}
• :hash - secure random-salt or explicit deterministic hashing
• :pseudonymize - reversible vault-backed tokenization
• :custom - application module implementing Obscura.Operator.Custom

Operator configurations are validated before any replacement is applied. Unknown operators, malformed options, callback failures, and missing vaults return a value-safe %Obscura.Anonymizer.Error{}. See docs/operators.md for the complete schemas and hash migration guidance.

Redact

{:ok, result} = Obscura.redact("Call 202-555-0188", entities: [:phone])

result.text
#=> "Call [PHONE]"

Structured inputs return %Obscura.Structured.Result{} instead of %Obscura.Anonymizer.Result{}:

input = %{email: "jane@example.com", password: "secret"}

{:ok, result} =
  Obscura.redact(input,
    entities: [:email],
    field_policies: %{password: :drop}
  )

result.data
#=> %{email: "[EMAIL]"}

Extensibility

Custom recognizer modules implement Obscura.Recognizer:

defmodule TicketRecognizer do
  @behaviour Obscura.Recognizer

  alias Obscura.Analyzer.Result

  def name, do: :ticket
  def supported_entities, do: [:ticket]

  def analyze(text, _opts) do
    for [{start, length}] <- Regex.scan(~r/TKT-\d{4}/, text, return: :index) do
      %Result{
        entity: :ticket,
        start: start,
        end: start + length,
        byte_start: start,
        byte_end: start + length,
        score: 0.8,
        text: binary_part(text, start, length),
        source_entity: "TICKET",
        recognizer: :ticket,
        metadata: %{}
      }
    end
  end
end

Obscura.analyze("Ticket TKT-1234",
  entities: [:ticket],
  recognizers: [TicketRecognizer]
)

Inline pattern definitions and deny lists are available without modifying Obscura source:

employee_id =
  Obscura.Recognizer.PatternDefinition.new!(
    name: :employee_id,
    entity: :employee_id,
    patterns: [%{name: :employee_id_v1, regex: ~r/EMP-\d{6}/, score: 0.65}],
    context: ["employee"]
  )

Obscura.analyze("employee EMP-123456",
  entities: [:employee_id],
  recognizers: [employee_id],
  context: ["employee"],
  explain: true
)

Obscura.analyze("Project ORCHID",
  entities: [:project_codename],
  deny_lists: [%{entity: :project_codename, values: ["orchid"], case_sensitive: false}]
)

Allow lists remove known-safe values after recognition:

Obscura.analyze("support@example.com jane@example.com",
  entities: [:email],
  allow_list: [%{entity: :email, values: ["support@example.com"]}]
)

Optional NER

Explicit NER/model-output support for open-class entities is opt-in and does not download or start models by default.

serving =
  Obscura.Recognizer.NER.FakeServing.new(%{
    "Alice works at Acme." => [
      %{label: "PER", start: 0, end: 5, score: 0.94},
      %{label: "ORG", start: 15, end: 19, score: 0.91}
    ]
  })

Obscura.analyze("Alice works at Acme.",
  entities: [:person, :organization],
  recognizers: [{Obscura.Recognizer.NER, serving: serving}]
)

Batch analysis preserves input order:

Obscura.analyze_many(["Alice", "Denver"],
  entities: [:person, :location],
  recognizers: [{Obscura.Recognizer.NER, serving: serving}]
)

Structs, Logger, and Plug

Structs can opt into field-aware redaction with @derive:

defmodule User do
  @derive {Obscura.Redactable,
           fields: [email: {:entity, :email}, password_hash: :drop, profile: :traverse]}
  defstruct [:email, :password_hash, :profile]
end

Obscura.redact(%User{email: "jane@example.com"}, entities: [:email])

Logger helpers redact metadata and inspected terms before they are handed to application logs:

Obscura.Logger.redact_metadata([user: "jane@example.com", password: "secret"],
  entities: [:email]
)

Obscura.Phoenix.Plug is Plug-compatible and can either assign redacted request fields or replace them:

plug Obscura.Phoenix.Plug,
  fields: [:params],
  mode: :assign_redacted,
  entities: [:email]

Telemetry events include durations, counts, entities, and statuses, but omit raw text and span values.

Vaults and Rehydration

Vault-backed pseudonymization supports reversible workflows:

{:ok, vault} = Obscura.Vault.Memory.start_link()

{:ok, result} =
  Obscura.redact("Email jane@example.com",
    entities: [:email],
    operators: %{email: %{type: :pseudonymize}},
    vault: vault
  )

result.text
#=> "Email <<EMAIL_001>>"

{:ok, original} = Obscura.rehydrate(result.text, vault: vault)
original
#=> "Email jane@example.com"

ETS-backed vaults are also available for explicitly supervised local sessions:

children = [
  {Obscura.Vault.ETS, name: MyApp.PiiVault}
]

Structured data can use the same operator:

{:ok, result} =
  Obscura.redact(%{message: "Email jane@example.com"},
    entities: [:email],
    operators: %{email: %{type: :pseudonymize}},
    vault: vault
  )

Obscura.rehydrate(result.data, vault: vault)

Vaults intentionally retain raw values in memory so rehydration can work. Applications should protect vault access and clear vaults when a request, chat, or support session no longer needs rehydration.

LLM Workflows

The LLM helpers operate on provider-independent message maps:

messages = [
  %{role: "system", content: "Be concise."},
  %{role: "user", content: "Email jane@example.com"}
]

{:ok, safe_messages, vault} =
  Obscura.LLM.redact_messages(messages, vault: :memory, entities: [:email])

safe_messages
#=> [
#=>   %{role: "system", content: "Be concise."},
#=>   %{role: "user", content: "Email <<EMAIL_001>>"}
#=> ]

Obscura.LLM.rehydrate_response("I will contact <<EMAIL_001>>.", vault: vault)

For streaming responses, use the streaming rehydrator:

{:ok, stream} = Obscura.Stream.Rehydrator.new(vault: vault)
{:ok, ready, stream} = Obscura.Stream.Rehydrator.feed(stream, "Hello <<EMA")
{:ok, ready2, stream} = Obscura.Stream.Rehydrator.feed(stream, "IL_001>>")
{:ok, rest} = Obscura.Stream.Rehydrator.flush(stream)

Evaluation

The fixture and evaluation harnesses are part of the library development workflow:

mix test
mix obscura.fixtures
mix obscura.eval --dataset synth_dataset_v2 --profile regex_only --smoke
mix obscura.eval --dataset synth_dataset_v2 --profile context --smoke
mix obscura.eval --dataset synth_dataset_v2 --profile llm_safe --smoke
mix quality
mix ci

Pinned Presidio-Research benchmark snapshots are committed under eval/datasets/presidio_research/ with checksums, provenance, and license attribution, so dependency-light evaluation works from a fresh clone.

Security And Privacy

Obscura is local and dependency-light. It does not include remote-provider recognizers, and model profiles never download assets during analyze/2 or redact/2. Reports, telemetry, diagnostics, and prediction exports omit raw detected values by default. Callers still own input logging, vault retention, credentials, and deployment controls. Applications can implement external integrations through the public custom-recognizer contract.

Vault pseudonymization is reversible and retains original values until the vault is cleared or stopped. Obscura does not currently provide encrypted persistent vault storage or compliance certification.

Maturity

Obscura is suitable for early-release integration and controlled evaluation. It is not yet a production 1.0, a compliance guarantee, or a complete Presidio replacement. Select a profile from measured dataset-specific evidence and read docs/known-limitations.md before deployment.

Current release-relevant evidence lives under eval/authoritative/. It includes two measured runs per stable profile and dataset across generated_large/template_heldout, synth_dataset_v2, and nemotron_pii_test_subset. Files generated under ignored eval/reports/ are historical or working evidence unless promoted by that manifest.

Obscura remains an early release. A stable profile means governed behavior and evidence, not universal production readiness or regulatory compliance.

Security

Obscura keeps errors, diagnostics, default inspection, telemetry, and authoritative reports value-safe, but its public results can contain raw or rehydrated values by design. Use include_text: false when detected source text is unnecessary, supervise and clear reversible vaults, and treat custom callbacks and optional model runtimes as trusted code.

Memory and ETS vaults are not encrypted persistent stores. Clearing or stopping a vault removes accessible mappings but cannot guarantee secure erasure of BEAM or native-runtime memory. See docs/security-threat-model.md, docs/known-limitations.md, and SECURITY.md.

Report suspected vulnerabilities privately through GitHub Private Vulnerability Reporting. Do not include raw PII, credentials, production vault contents, or private datasets in a report.

Compatibility

Patch releases preserve stable contracts. Stable breaking changes require at least one subsequent minor release and 90 days of deprecation, except for an urgent security or data-corruption fix. Human-readable error text and metadata contents are not stable; branch on documented codes and fields.