search_core

Multilingual full-text search building blocks for plain Elixir/Ecto — no Ash required. Two pieces, both driven by the pure-Elixir text_stemmer (Snowball, 33 languages, no NIF and no toolchain to install):

SearchCore.Pipeline — turn raw text into normalized, stemmed tokens.
SearchCore.Tsvector — turn those tokens into the search_text you index and the tsquery you search with, both from the same pipeline so index and query never drift apart.

Part of the search_ash monorepo.

SearchCore.process("Les idées mangent les chevaux", :fr)
#=> ["ide", "mangent", "cheval"]

SearchCore.searchable_text("Les chevaux mangent", :fr)  #=> "cheval mangent"
SearchCore.tsquery("chevaux", :fr)                      #=> "cheval"

Naming a language

Languages are ISO 639-1 codes (:fr, :en, :de) — one spelling per language, no aliases:

SearchCore.Language.supported_languages()  # the 36 codes, one per algorithm
SearchCore.Language.supported?(:fr)        #=> true
SearchCore.Language.validate!(:klingon)    # ** (ArgumentError) unsupported language …

text_stemmer is the authority

Which languages exist is asked of text_stemmer at call time, never frozen in. Upgrading text_stemmer makes any language it adds usable immediately — no search_core release, no recompile, nothing to maintain here. search_core deliberately carries no table of language names that could disagree with the stemmer.

Need a human-readable, localized language name for a UI? That's what the ISO code is for — hand it to ex_cldr_languages, which does it properly in every locale.

Algorithm variants

Snowball ships more than one algorithm for a couple of languages. Those are addressed by code only (:en_porter, :en_lovins, :nl_porter), while :english and :dutch map to the current recommended algorithm. A variant still belongs to its language — :en_porter takes the English stopword list — which is what SearchCore.Language.base/1 gives you:

SearchCore.Language.base(:en_porter)  #=> :en

Postgres wiring

Tokens are already stemmed and accent-folded, so use the 'simple' config:

# indexing — store this in a column, GIN-index its tsvector
searchable = SearchCore.searchable_text(title <> " " <> body, :fr)

# querying
import Ecto.Query
tsquery = SearchCore.tsquery(user_input, :fr)

from a in Article,
  where: fragment("to_tsvector('simple', ?) @@ to_tsquery('simple', ?)", a.search_text, ^tsquery)

Because both sides go through the same pipeline, a search for "chevaux" finds a row that stored "cheval" — no "search returns nothing" surprises.

See examples/basic for a runnable Ecto demo (multilingual seed, search, and an EXPLAIN confirming GIN index usage).