LemonCore

Foundational shared library for the Lemon umbrella project. All other apps depend on lemon_core -- it provides configuration management, configured-implementation contract validation, encrypted secrets, pluggable storage, an event bus, live named-node invocation routing, session routing primitives, idempotency, execution approvals, telemetry, and quality tooling.

LemonCore.Context is the shared versioned preview/resolve boundary for root-confined file and folder references, shell-free git diffs, SSRF-guarded public URLs, redacted session exports, and format-sniffed document text. See docs/user-guide/context-references.md.

This app has zero dependencies on other umbrella apps and must remain that way.

Architecture Overview

                   +-----------------------+
                   |    LemonCore.Config   |  TOML parsing, env overrides,
                   |  .Modular / .Providers|  hot reload, validation
                   +-----------+-----------+
                               |
                   +-----------+-----------+
                   |  LemonCore.ConfigCache |  ETS cache with mtime-based
                   |  LemonCore.ConfigReloader| invalidation + FileSystem watcher
                   +-----------+-----------+
                               |
          +--------------------+--------------------+
          |                    |                    |
+---------+--------+ +--------+--------+ +---------+--------+
| LemonCore.Secrets| | LemonCore.Store | | LemonCore.Bus    |
| .Crypto/.External| | .EtsBackend     | | (Phoenix.PubSub) |
| .Source/.Runner  | | .SqliteBackend  | |                  |
| .MasterKey       | | .JsonlBackend   | | LemonCore.Event  |
+------------------+ | .ReadCache      | +------------------+
                     +-----------------+
                               |
          +--------------------+--------------------+
          |                    |                    |
+---------+--------+ +--------+--------+ +---------+--------+
| LemonCore.       | | LemonCore.      | | LemonCore.       |
| Idempotency      | | ExecApprovals   | | Introspection    |
+------------------+ +-----------------+ +------------------+
          |
+---------+--------+
| LemonCore.       |
| RouterBridge     |  Runtime bridge to :lemon_router
| SessionKey       |  without compile-time coupling
| RunRequest       |
| InboundMessage   |
+------------------+

Supervised Process Tree

LemonCore.Application starts these children under a :one_for_one supervisor:

Durable memory moved to lemon_memory, the workspace stores to agent_core, and provider credential-pool rotation to lemon_agent (LemonAgent.ModelRuntime.ProviderPoolRotator); each supervises its own processes.

Browser, media-job, and LSP drivers live in lemon_browser, lemon_media, and lemon_lsp. Core doctor diagnostics may probe them at runtime, but lemon_core must not depend on those apps.

Module Inventory

Configuration

Doctor and Support

mix lemon.doctor includes usage.status, which reads the same redacted aggregate usage diagnostics and warns when configured run, token, or cost limits are exceeded.

Checks for domains owned by other apps live in those apps and register through config :lemon_core, :doctor_checks (memory: LemonMemory.Doctor.Checks.Memory; skills: LemonSkills.Doctor.Checks.Skills; channels and automation likewise).

Secrets

LemonCore.Redact is the common credential policy: text/1 masks assignments, Bearer/Basic authorization, provider keys, JWTs, and PEM private keys; map/1 also masks sensitive-key values recursively through maps, structs, lists, and tuples. Structs become plain maps. sensitive_key?/1 accepts atom/string keys. Redaction is idempotent and does not truncate output.

Use text(text, count: true) for {redacted_text, replacement_count}, drop_keys: true when secret names must also be omitted, and strict: true for portable profile/config exports that also omit full sensitive assignment lines and unlabelled hex keys. Keep caller-specific content allowlists, URL/path omission, collection/depth/byte bounds, and bounded error summaries: credential redaction alone does not make arbitrary data safe to export. Redact before truncating so partial credentials cannot escape.

Storage

Memory

Durable memory moved to the lemon_memory app (LemonMemory.*). Core reaches it only through the LemonCore.Store finalize-run hook the runtime configures.

Event System

Routing and Sessions

Operations

Utilities

Quality

Mix Tasks

Provider onboarding preferred_models should favor stable models that actually work on Lemon's live transport path. Do not keep known-broken variants in those default suggestions just because the upstream provider documents them. | mix lemon.onboard.anthropic | Anthropic provider setup | | mix lemon.onboard.antigravity | Antigravity (Google) provider setup with OAuth | | mix lemon.onboard.gemini | Google Gemini CLI / Code Assist setup with OAuth | | mix lemon.onboard.codex | OpenAI Codex provider setup with OAuth and localhost callback capture | | mix lemon.onboard.copilot | GitHub Copilot provider setup with OAuth | | mix lemon.quality | Run all quality checks | | mix lemon.store.migrate_jsonl_to_sqlite | Migrate JSONL to SQLite | | mix lemon.introspection | Query introspection events | | mix lemon.check_duplicate_tests | Check for duplicate test names |

Configuration System

Config Sources (precedence: env > project > global)

1. Global: ~/.lemon/config.toml
2. Project: <cwd>/.lemon/config.toml
3. Environment: LEMON_* and provider-specific variables override both

Config Sections

• providers -- LLM API keys and base URLs (anthropic, openai, openai-codex, opencode, kimi, google)
• defaults -- Preferred home for default provider/model/thinking level/engine
• runtime -- Runtime behavior (compaction, retry, shell, tools, cli, extensions, theme)
• profiles -- Per-agent identity/model/node defaults and tool policies. The user-managed lifecycle lives in LemonCore.ProfileStore, with derived homes at ~/.lemon/profiles/<id>/ and stable agent:<id>:main session keys.
• agent -- Legacy alias for runtime/default settings (still supported)
• agents -- Legacy alias for profile settings (still supported)
• tui -- Theme, debug mode
• logging -- File logging, level, rotation
• gateway -- Max concurrent runs, engine bindings, SMS/email/webhook settings, plus per-platform [gateway.<id>] sections resolved by their own app (see LemonCore.Config.Gateway.Channel)

Access Patterns

# Cached read (default hot path) - uses ConfigCache when available
config = LemonCore.Config.load(cwd)

# Force reload from disk (updates cache)
config = LemonCore.Config.reload(cwd)

# Access nested values
provider = LemonCore.Config.get(config, [:agent, :default_provider], "anthropic")

# Modular config interface (typed sub-structs, validation)
config = LemonCore.Config.Modular.load(project_dir: cwd)
config = LemonCore.Config.Modular.load!(project_dir: cwd)
{:ok, config} = LemonCore.Config.Modular.load_with_validation(project_dir: cwd)

Hot Reload Flow

LemonCore.ConfigReloader.reload/1
  |-- Acquire reload lock
  |-- Compute file/env/secrets digests
  |-- Compare with cached digests
  |-- Reload .env if changed
  |-- Reload TOML via LemonCore.Config.reload/2
  |-- Compute redacted diff vs previous snapshot
  |-- Update ConfigCache
  |-- Broadcast :config_reloaded on "system" topic
  |-- On failure: keep last good snapshot, emit :config_reload_failed

First-run readiness

LemonCore.Setup.Readiness.status/1 derives one shared readiness state from the global config, secrets backend, default provider/model, and credential source. It never performs a network request or mutates setup state. Clients use pending_steps/1 and ready?/1 so a setup wizard, first-run gate, or browser cannot silently invent a different definition of “ready.” Live provider verification remains in lemon_cli and doctor diagnostics.

Environment Variable Overrides

Secrets Management

Storage Model

Secrets are encrypted at rest with AES-256-GCM. Per-secret encryption keys are derived via HKDF-SHA256 from a master key and a random 32-byte salt. The ciphertext, nonce, and salt are base64-encoded and stored in the :secrets_v1 Store table.

Master Key Resolution (in order)

1. macOS Keychain (Lemon Secrets service, default account) -- preferred
2. LEMON_SECRETS_MASTER_KEY environment variable -- fallback
3. ~/.lemon/secrets_master_key file -- local non-keychain fallback
4. Fail with :missing_master_key

The order and the locations are configurable via config :lemon_core, LemonCore.Secrets, key_providers: [:keychain, :env, :file], key_file: ..., env_var: ...; see LemonCore.Secrets.KeyProvider. The key must be base64-encoded 32 bytes.

For local Linux/dev usage, treat ~/.lemon/secrets_master_key as the canonical master key file. ./bin/lemon normalizes LEMON_SECRETS_MASTER_KEY from that file before boot so stale inherited shell env does not override the working local key.

To change the master key without re-entering secrets, run lemon secrets rotate-key (or mix lemon.secrets.rotate_key): it previews, then with the preview's --confirm token re-encrypts every stored secret under a new key and writes that key to the Keychain entry or key file that supplied the old one. See LemonCore.Secrets.KeyRotation and the secrets contract.

API

# Store a secret
{:ok, metadata} = LemonCore.Secrets.set("api_key", "secret_value", provider: "manual")

# Retrieve a secret
{:ok, value} = LemonCore.Secrets.get("api_key")

# Resolve (store first, then external sources, then env fallback)
{:ok, value, :store} = LemonCore.Secrets.resolve("api_key")
{:ok, value, "external:command:local_helper"} = LemonCore.Secrets.resolve("api_key")
{:ok, value, :env} = LemonCore.Secrets.resolve("MISSING_FROM_STORE")

# Convenience (returns value or nil)
value = LemonCore.Secrets.fetch_value("ANTHROPIC_API_KEY")

# Check existence
exists? = LemonCore.Secrets.exists?("api_key")

# List (metadata only, no values)
{:ok, metadata_list} = LemonCore.Secrets.list()

# Delete
:ok = LemonCore.Secrets.delete("api_key")

# Status
status = LemonCore.Secrets.status()

External sources are read-only fallbacks behind the encrypted store and ahead of environment variables. They are disabled unless their config contains the exact boolean enabled = true. Enabled-source failures stop resolution before environment fallback; a source that succeeds without the requested name may continue. Use env_fallback: false to disable both external and environment fallbacks. Secret reads update usage metadata (usage_count, last_used_at) without mutating updated_at.

1Password, Bitwarden Secrets Manager, and arbitrary argv-only command adapters share one supervised runner with exact config validation, minimal child environments, stable redacted errors, and configured time/output limits. The optional bounded process-local cache defaults off. Inspect or live-test them without revealing values:

lemon secrets sources status --json
lemon secrets sources test [source-id] --json

See docs/config.md for the exact source schemas and bootstrap rules.

LemonCore.Secrets.EnvCatalog.names/0 is the canonical ordered set used by the packaged lemon secrets check / lemon secrets import-env commands and their Mix equivalents. It is intentionally separate from the release-profile runtime declarations in LemonCore.Env.

Storage Backends

When to Use Each

Configuration

# config/config.exs
config :lemon_core, LemonCore.Store,
  backend: LemonCore.Store.SqliteBackend,
  backend_opts: [path: "/var/lib/lemon/store.sqlite3"]

Wrapper-First Table Access

defmodule MyApp.WidgetStore do
  @table :widgets

  def put(id, widget), do: LemonCore.Store.put(@table, id, widget)
  def claim(id, widget), do: LemonCore.Store.put_new(@table, id, widget)
  def get(id), do: LemonCore.Store.get(@table, id)
end

MyApp.WidgetStore.put(id, widget)
widget = MyApp.WidgetStore.get(id)

put_new/3 is the insert-if-absent primitive for durable claims. It returns :ok for the first writer and {:error, :exists} for later writers without overwriting the original value. take/2 atomically consumes a key, compare_and_swap/4 replaces a value only when its exact expected value is still current, and compare_and_delete/3 conditionally removes an exact snapshot so a sweeper cannot erase a renewed lease or receipt. compare_and_delete_many/1 first validates a set of exact snapshots in one serialized Store call. SQLite commits the set transactionally, ETS applies it under the Store owner, and ordered backends report the first failure so callers can place a safety-critical durable fence last.

Store calls are fail-soft: if the GenServer is overloaded/unavailable, write APIs return {:error, :store_unavailable} and ordinary read/list APIs return nil/[]. Correctness-sensitive callers use fetch/3 and fetch_all/2 to distinguish confirmed absence or an empty table from an unavailable backend. Store diagnostics hash keys for idempotency tables so caller replay secrets do not enter logs.

Use the generic table API only inside wrapper modules, backend internals, or explicitly app-local legacy tables. Shared-domain runtime code should go through typed wrappers so table ownership stays localized. Wrappers are plain modules over the generic API; LemonCore.Store has no domain-specific chat-state, policy, or progress functions.

Specialized APIs

Shared-domain callers should prefer typed wrappers:

• Chat state: LemonCore.ChatStateStore.put/2, get/1, delete/1 (each also takes a store name first); stamps a 24h :expires_at on write, treats expired entries as absent on read, and the store's periodic sweep removes expired :chat entries
• Run history: LemonCore.RunStore.append_event/2, finalize/2, history/2, get/1
• Policies: LemonCore.PolicyStore.put_agent/2, put_channel/2, put_session/2, put_runtime/1
• Idempotency: LemonCore.IdempotencyStore.put/3, get/2, delete/2
• Introspection: LemonCore.IntrospectionStore.append/1, list/1
• Project bindings: LemonCore.ProjectBindingStore.get_override/1, put_override/2, get_dynamic/1
• Exec approvals: LemonCore.ExecApprovalStore.get_pending/1, put_pending/2, policy getters/setters by scope

RunStore.finalize/2 returns :ok only after the immutable final summary and its session index are written. Retrying the same summary repairs a partial index failure without double-counting the run: the session entry remembers its last 32 indexed run ids, so its size stays fixed however many runs the session has. Finalize hooks run after both writes and use at-least-once delivery, so hook implementations must be idempotent by run id.

Agent workspace coordination — goals and kanban boards — is built on this Store but lives in lemon_agent as LemonAgent.Workspace.{GoalStore, KanbanStore}. Heartbeats use automation cron jobs and cron history.

ReadCache

LemonCore.Store.ReadCache maintains public ETS tables for hot domains (:runs, plus the default cached tables :sessions_index, :chat, and :progress, plus tables collaborators add with register_cached_table/1). Reads bypass the GenServer entirely for O(1) ETS lookup, while writes go through the GenServer which updates both the backend and cache atomically.

Backend Behaviour

Implementing a new backend requires the LemonCore.Store.Backend behaviour:

@callback init(opts()) :: {:ok, state()} | {:error, term()}
@callback put(state(), table(), key(), value()) :: {:ok, state()}
@callback put_new(state(), table(), key(), value()) :: {:ok, state()} | {:exists, state()} | {:error, term()}
@callback get(state(), table(), key()) :: {:ok, value() | nil, state()}
@callback delete(state(), table(), key()) :: {:ok, state()}
@callback list(state(), table()) :: {:ok, [{key(), value()}], state()}

SQLite serializes keys and values with :erlang.term_to_binary/1. The JSONL backend uses a JSON codec that preserves atoms, tuples, structs, and nested map keys for portable append-only files.

Event Bus

Standard Topics

Usage

# Subscribe
LemonCore.Bus.subscribe("run:" <> run_id)

# Receive
receive do
  %LemonCore.Event{type: :delta, payload: payload} -> handle_delta(payload)
  %LemonCore.Event{type: :completed, payload: payload} -> handle_completion(payload)
after
  30_000 -> :timeout
end

# Broadcast
event = LemonCore.Event.new(:run_started, %{engine: "lemon"}, %{run_id: run_id})
LemonCore.Bus.broadcast("session:" <> session_key, event)

# Unsubscribe
LemonCore.Bus.unsubscribe("run:" <> run_id)

Session Keys

Session keys provide stable identifiers for routing and state management.

Formats

• Main: agent:<agent_id>:main[:sub:<sub_id>]
• Channel: agent:<agent_id>:<channel_id>:<account_id>:<peer_kind>:<peer_id>[:thread:<thread_id>][:sub:<sub_id>]

Usage

key = LemonCore.SessionKey.main("my_agent")
# => "agent:my_agent:main"

key = LemonCore.SessionKey.channel_peer(%{
  agent_id: "my_agent", channel_id: "demo",
  account_id: "bot123", peer_kind: :dm, peer_id: "user456"
})

%{agent_id: id, kind: :main} = LemonCore.SessionKey.parse(key)

Execution Approvals

Tool execution gating with scope-based persistence. Placed in lemon_core so any app can request/resolve approvals without depending on :lemon_router.

Scopes

# Request (blocks until resolved)
case LemonCore.ExecApprovals.request(%{
  run_id: run_id, session_key: session_key,
  tool: "shell", action: %{command: "rm -rf /tmp/old"},
  expires_in_ms: 60_000
}) do
  {:ok, :approved, scope} -> proceed()
  {:ok, :denied} -> halt()
  {:error, :timeout} -> handle_timeout()
end

# Resolve (called by UI/admin). Atomic: a racing cancel/resolve on the same
# id has exactly one winner; the loser gets {:error, :not_pending}.
:ok = LemonCore.ExecApprovals.resolve(approval_id, :approve_session)

Approval requests, resolutions, and timeouts also write redacted approval_* introspection events with approval id, tool, action type/hash, and decision/scope metadata. Raw action payloads, prompts, and secrets are not stored in the timeline. Timeouts also broadcast :approval_resolved with decision: :timeout so operator clients can clear stale pending requests live.

Idempotency

At-most-once deduplication backed by the Store with 24h TTL.

result = LemonCore.Idempotency.execute("messages", msg_id, fn ->
  perform_operation()
end)

Introspection

Canonical event persistence with redaction for tool arguments and sensitive fields.

:ok = LemonCore.Introspection.record(:tool_completed,
  %{tool_name: "exec", result_preview: "ok"},
  run_id: run_id, session_key: session_key, engine: "codex"
)

events = LemonCore.Introspection.list(run_id: run_id, limit: 50)

RouterBridge

Channel adapters forward runs to :lemon_router without compile-time coupling. The router registers itself at startup.

{:ok, run_id} = LemonCore.RouterBridge.submit_run(run_request)
:ok = LemonCore.RouterBridge.handle_inbound(inbound_message)
:ok = LemonCore.RouterBridge.abort_session(session_key, :user_requested)
{:ok, busy?} = LemonCore.RouterBridge.session_busy?(session_key)
{:ok, active_run_id} = LemonCore.RouterBridge.active_run(session_key)
{:ok, sessions} = LemonCore.RouterBridge.list_active_sessions()

An unregistered router returns {:error, :unavailable}. A configured mutating callback that raises, throws, returns a malformed acknowledgement, times out, or exits returns {:error, :outcome_unknown}: even a :noproc exit may follow an earlier side effect inside a multi-step callback. Callers must not retry automatically without independent idempotency or reconciliation. Read-only query exits and throws remain unavailable because they cannot duplicate a side effect. Query calls do not substitute false, :none, or []; callers must handle unknown router state explicitly. Query exceptions return the sanitized {:error, :query_failed} rather than carrying exception messages across the bridge.

Configured implementations are validated against the bridge behaviours before use. submit_run/1 accepts only {:ok, nonempty_binary_run_id} as success, and an explicit {:error, reason} other than {:error, :outcome_unknown} means the implementation guarantees the mutation did not take effect. Malformed mutation answers are outcome-unknown; malformed query answers remain {:error, {:unexpected_answer, value}}. Active-session lists are accepted only when every entry has non-empty binary session_key and run_id fields. For abort and keep-alive commands, :ok acknowledges that the router accepted or dispatched the decision; it does not wait for synchronous application by the target run process. Failure logs contain only a sanitized callback MFA and failure class; raw exceptions, exit reasons, stacktraces, and callback arguments are neither logged nor returned for query exceptions.

Telemetry Events

Dependencies

Everything else is optional — lemon_core boots without it, with the degradation noted below. UUIDs are generated by the vendored LemonCore.UUID.

If you want run history, declare exqlite yourself

exqlite being optional has a consequence that is easy to miss: LemonCore.Application only starts LemonCore.RunHistoryStore when the SQLite NIF is loadable. An application that depends on lemon_core but does not itself declare exqlite boots fine, but run history is disabled — get_run_history/3 returns [] and the finalize-run hooks that persist history do nothing. The only signal is a warning at boot.

Optional dependencies are not inherited: add it to your own deps/0 to enable it.

{:exqlite, "~> 0.34"}

Durable memory has the same requirement in lemon_memory.

What the Hex package leaves out

The Hex tarball ships the library, not this repository's tooling: the architecture/docs lint (LemonCore.Quality.*, mix lemon.quality, mix lemon.architecture.docs, mix lemon.check_duplicate_tests), the managed-install self-updater (LemonCore.Update.*, mix lemon.update), the platform benchmark runner (mix lemon.bench) and the repo task catalog (mix lemon.help) exist only in the lemon checkout.

Filesystem layout

Everything lemon_core reads or writes by convention hangs off a per-user state directory (~/.lemon) and a per-project one (<project>/.lemon). The defaults are the reference runtime's; LemonCore.Paths makes them configurable so a host application can put its state somewhere not named after Lemon:

config :lemon_core, :paths,
  state_dir: ".lemon",          # per-user and per-project directory name
  config_file: "config.toml",   # file inside it
  home_state_dir: nil,          # pin the per-user directory outright
  global_config: nil            # pin the global config file outright

Renaming :state_dir moves everything derived from it: both config files, the config reloader's watched paths, the default run-history store directory, and the doctor's extension directory. Two locations are configured separately because they are not part of that layout: the secrets master key (config :lemon_core, LemonCore.Secrets, key_file: ...) and the store backend path (config :lemon_core, LemonCore.Store, backend_opts: [path: ...]).

Testing

# Run all lemon_core tests
mix test apps/lemon_core

# Run specific test file
mix test apps/lemon_core/test/lemon_core/config_test.exs

# Run with coverage
mix test --cover apps/lemon_core

Test Harness

defmodule MyTest do
  use LemonCore.Testing.Case, async: true

  test "example", %{harness: harness, tmp_dir: tmp_dir} do
    path = temp_file!(harness, "config.toml", "[agent]\ndefault_model = \"test\"")
    key = unique_session_key("mytest")
  end
end

Available helpers: unique_token/0, unique_scope/0, unique_session_key/0, unique_run_id/0, temp_file!/3, temp_dir!/2, clear_store_table/1, mock_home!/1, random_master_key/0.

Important Notes

• Never add umbrella app dependencies to lemon_core -- it is the base layer
• Keep module interfaces stable -- other apps depend on them
• LemonCore.Config.load/2 uses the cache by default; LemonCore.Config.reload/2 forces a disk read
• Secret values are never logged or returned by list/status APIs
• SQLite store values serialize with :erlang.term_to_binary/1; JSONL uses a JSON codec with Elixir-term markers for portable persistence
• Events use millisecond timestamps from System.system_time(:millisecond)
• RouterBridge returns {:error, :unavailable} when :lemon_router has not registered, and {:error, :outcome_unknown} when a configured mutation loses its acknowledgement; never automatically retry the latter
• Dedupe.Ets uses monotonic time for TTL; Idempotency uses wall-clock time
• Config.Modular is the newer typed approach; Config is still the primary interface