Yargy

Earley parser with grammar DSL for rule-based information extraction from Russian text.

Elixir port of natasha/yargy. Depends on morph_ru for morphological analysis and razdel for sentence segmentation.

• Earley parser optimized for Russian free word order
• Declarative grammar DSL with compile-time caching
• 18 predicates: gram/1, type/1, eq/1, in_/1, normalized/1, dictionary/1, …
• Relations for morphological agreement (gender-number-case, number-case)
• Interpretation system for structured output (facts)
• Tokenizer, sentence splitter, morph tagger, morph pipeline
• Built-in grammars: Date, NamedDate, Amount, Person

Installation

def deps do
  [
    {:yargy, "~> 0.5"}
  ]
end

Performance

Benchmarked on real Russian legal text (2.5 KB medium, 18.5 KB long). Person grammar extracts full names with patronymics in four patterns (Surname+Name+Patronymic, initials, etc.).

End-to-end: text → extracted matches (tokenize + morph tag + parse):

Component breakdown (long text, 18.5 KB, ~3600 tokens):

Python's tokenizer is backed by C-compiled regex + pymorphy2's DAWG in C, giving it an edge on raw tokenization. The Elixir port is near parity on tokenize+morph and wins on the parser core where Earley chart operations dominate — tuple-based O(1) term access, integer rule IDs for fast dedup, and prediction caching reduce constant factors.

Run benchmarks: mix run bench/parser_bench.exs (Elixir), python3 bench/python_bench.py (Python).

Grammar DSL

Define grammars declaratively with use Yargy.Grammar. Rules compile once at module load time and are cached in persistent_term.

defmodule MyApp.PersonGrammar do
  use Yargy.Grammar

  defrule :surname, all([gram("Surn"), capitalized()])
  defrule :first_name, all([gram("Name"), capitalized()])
  defrule :patronymic, all([gram("Patr"), capitalized()])
  defrule :dot, token(".")
  defrule :initial, all([upper(), length_eq(1)])
  defrule :initial_dot, rule(:initial) ~> rule(:dot)

  defgrammar :person, choice([
    rule(:surname) ~> rule(:first_name) ~> optional(rule(:patronymic)),
    rule(:first_name) ~> optional(rule(:patronymic)) ~> rule(:surname),
    rule(:surname) ~> rule(:initial_dot) ~> rule(:initial_dot),
    rule(:initial_dot) ~> rule(:initial_dot) ~> rule(:surname)
  ])
end

# Use the generated functions:
MyApp.PersonGrammar.person_text("Адвокат Иванов Иван Петрович подписал")
# [%Yargy.Parser.Match{tokens: [...], start: 1, stop: 4}]

Terminals

Composition

Generated functions

defgrammar :person, ... generates:

• person(tokens) — find all matches in morph-tagged tokens
• person_text(text) — tokenize + morph-tag, then find matches
• person_parser() — return the pre-built %Parser{} struct

Bag-of-features matchers

defmatch checks for unordered token presence — no sequential parsing needed. Useful for sentence classification.

defmodule MyApp.SentenceClassifier do
  use Yargy.Grammar

  defmatch :evidence, all_of([
    any_token(lemma(~w[подтверждаться подтвердить])),
    any_token(lemma(~w[акт квитанция чек выписка]))
  ])

  defmatch :demand, any_token(all([
    lemma(~w[требовать просить взыскать]),
    gram("VERB")
  ]))

  defmatch :short_title, all_of([
    any_token(lemma("претензия")),
    max_words(5)
  ])
end

MyApp.SentenceClassifier.evidence_match?("Оплата подтверждается актом")
# true

MyApp.SentenceClassifier.demand_match?("Истец требует возмещения")
# true

Match combinators

Generated functions

defmatch :evidence, ... generates:

• evidence?(tokens) — check morph-tagged tokens
• evidence_match?(text) — tokenize + morph-tag, then check

Low-level API

The DSL compiles to the same Rule / Predicate / Parser primitives you can use directly:

alias Yargy.{Parser, Predicate, Rule, Tokenizer}

date = Rule.rule([
  Predicate.type(:int), Predicate.eq("."),
  Predicate.type(:int), Predicate.eq("."),
  Predicate.type(:int)
])

tokens = Tokenizer.tokenize("Договор от 15.03.2024 подписан.")
Parser.findall(Parser.new(date), tokens)

Partial matching (autocomplete)

When the input ends mid-parse, partial_matches/2 returns incomplete states from the Earley chart — useful for autocomplete.

rule = Rule.rule([
  Predicate.in_(~w(ООО ПАО АО)),
  Predicate.or_([Predicate.eq("«"), Predicate.eq("\"")]),
  Predicate.type(:word),
  Predicate.or_([Predicate.eq("»"), Predicate.eq("\"")])
])

parser = Parser.new(rule)
tokens = Tokenizer.tokenize("Директору ООО «Ромашка")

Parser.partial_matches(parser, tokens)
# [%{dot: 3, production_length: 4, progress: 0.75,
#    matched_text: "ООО « Ромашка", ...}]

Sentence splitting

Yargy.Sentenize.sentenize("Привет. Как дела? Хорошо!")
# ["Привет.", "Как дела?", "Хорошо!"]

Core modules

License

MIT © Danila Poyarkov