Lotus Elasticsearch

Elasticsearch and OpenSearch source adapter for Lotus. Run Lotus queries, dashboards, and AI-assisted exploration against an Elasticsearch cluster the same way you would against Postgres, MySQL, or SQLite.

A Lotus statement here is an Elasticsearch Query DSL JSON object rather than SQL text. Variables, optional clauses, filters, sorts and pagination all rewrite that object structurally — see Writing Queries.

0.1.0 — first release. The adapter is complete against the Lotus v1 Lotus.Source.Adapter contract and its test suite runs against a live cluster, but its own surface — the client macro, the config shape, the index and field mapping, the type mapping — has no production users yet and may still move. Pin accordingly.

Installation

Add both lotus and lotus_elasticsearch to your mix.exs:

def deps do

  [

    {:lotus, "~> 1.0"},

    {:lotus_elasticsearch, "~> 0.1"}

  ]

end

Requires Elixir 1.18 or later. CI runs the suite on Elixir 1.18, 1.19 and 1.20 against OpenSearch 2.18.

Configuring a data source

There are two ways to put an Elasticsearch cluster into :data_sources. They are alternatives — pick one per source, not both. Both work because can_handle?/1 claims two kinds of entry: an atom that is a module exporting __elasticsearch__/0, and a map with adapter: :elasticsearch.

Option A — a client module

Define a module with the Lotus.Elasticsearch macro and let it read its connection settings from your application config:

defmodule MyApp.SearchClient do

  use Lotus.Elasticsearch, otp_app: :my_app

end

config :my_app, MyApp.SearchClient,

  url: "http://localhost:9200",

  username: "elastic",

  password: "changeme"



config :lotus,

  source_adapters: [Lotus.Source.Adapters.Elasticsearch],

  data_sources: %{

    "postgres" => MyApp.Repo,

    "search"   => MyApp.SearchClient

  }

The macro defines exactly three functions on your module: __elasticsearch__/0 (the marker can_handle?/1 looks for), config/0 (reads Application.get_env(otp_app, __MODULE__, [])), and url/0. wrap/2 calls config/0 and reads :url, :username and :password out of it.

Pick this when your connection settings come from config/runtime.exs and environment variables, or when you want one named module to refer to the cluster from elsewhere in your app. Application config is evaluated at runtime on every config/0 call, so a secret does not have to be baked into the :lotus config map.

Option B — an inline config map

config :lotus,

  source_adapters: [Lotus.Source.Adapters.Elasticsearch],

  data_sources: %{

    "postgres" => MyApp.Repo,

    "search"   => %{

      adapter: :elasticsearch,

      url: "http://localhost:9200",

      username: "elastic",

      password: "changeme",

      allow_unrestricted_resources: true

    }

  }

Pick this when you want everything about the source in one place, and especially when you want the per-source allow_unrestricted_resources opt-in described under Visibility — Lotus reads that key off the :data_sources entry itself, so it is only available on the map form. With Option A your only lever is the global flag.

Why :elasticsearch and not the adapter module

Lotus's canonical map form is %{adapter: SomeModule, ...}, which is used directly with no probing. The bare :elasticsearch atom here is deliberate and is not that form: core distinguishes the two by whether the atom is Elixir.-prefixed, and a non-prefixed atom is treated as the host application's own discriminator. The entry therefore falls through to can_handle?/1 probing, which is why :source_adapters must list Lotus.Source.Adapters.Elasticsearch for either option to resolve.

If you prefer the no-probing path, %{adapter: Lotus.Source.Adapters.Elasticsearch, url: ...} also works and lets you drop the :source_adapters entry — wrap/2 reads the same :url / :username / :password keys either way.

Running a query

{:ok, result} =

  Lotus.run_statement(

    ~s|{"query": {"term": {"status": "active"}}, "size": 10}|,

    [],

    repo: "search",

    index: "users"

  )

:index is an adapter-specific option that Lotus passes through untouched to execute_query/4; it becomes the index path segment of the _search call. It defaults to "_all", which searches every index in the cluster — almost never what you want, so pass it explicitly.

Two consequences worth knowing before you build on this:

What works

What this adapter cannot do

Being explicit about the SQL-shaped things that are absent:

Visibility

Unlike SQL adapters, where Lotus can statically parse which tables a statement touches, Elasticsearch queries name their indices in the HTTP path, not the JSON body. extract_accessed_resources/2 therefore returns {:unrestricted, reason}, and needs_preflight?/2 is always true — so out of the box every query against this source is blocked by preflight, with an error telling you how to opt in.

Opt in for the one source, on the map form:

config :lotus,

  data_sources: %{

    "search" => %{adapter: :elasticsearch, url: "...", allow_unrestricted_resources: true}

  }

or globally, which is the only option if you configured the source as a client module:

config :lotus, :allow_unrestricted_resources, true

The per-source value wins over the global flag in both directions. Opting in means you are trusting Elasticsearch's own index-level security to enforce who can read what — Lotus stops asking.

builtin_denies/1 does still hide dot-prefixed system indices (.kibana, .security, …) from the schema explorer and table-level deny checks. That is a listing rule, not an execution guard: it does not stop a statement whose :index option names one.

Development

mix deps.get

mix test.setup   # docker compose up -d — OpenSearch on port 9209

mix test

Unit tests run without a cluster. The tests under test/integration/ need the docker-compose cluster on http://localhost:9209; they create and tear down indices prefixed lotus_test.

Guides

License

MIT — see LICENSE.