LemonCore

Foundational shared library for the Lemon umbrella project. All other apps depend on lemon_core -- it provides configuration management, encrypted secrets, pluggable storage, an event bus, session routing primitives, idempotency, execution approvals, telemetry, and quality tooling.

This app has zero dependencies on other umbrella apps and must remain that way.

Architecture Overview

                   +-----------------------+
                   |    LemonCore.Config   |  TOML parsing, env overrides,
                   |  .Modular / .Providers|  hot reload, validation
                   +-----------+-----------+
                               |
                   +-----------+-----------+
                   |  LemonCore.ConfigCache |  ETS cache with mtime-based
                   |  LemonCore.ConfigReloader| invalidation + FileSystem watcher
                   +-----------+-----------+
                               |
          +--------------------+--------------------+
          |                    |                    |
+---------+--------+ +--------+--------+ +---------+--------+
| LemonCore.Secrets| | LemonCore.Store | | LemonCore.Bus    |
| .Crypto          | | .EtsBackend     | | (Phoenix.PubSub) |
| .Keychain        | | .SqliteBackend  | |                  |
| .MasterKey       | | .JsonlBackend   | | LemonCore.Event  |
+------------------+ | .ReadCache      | +------------------+
                     +-----------------+
                               |
          +--------------------+--------------------+
          |                    |                    |
+---------+--------+ +--------+--------+ +---------+--------+
| LemonCore.       | | LemonCore.      | | LemonCore.       |
| Idempotency      | | ExecApprovals   | | Introspection    |
+------------------+ +-----------------+ +------------------+
          |
+---------+--------+
| LemonCore.       |
| RouterBridge     |  Runtime bridge to :lemon_router
| SessionKey       |  without compile-time coupling
| RunRequest       |
| InboundMessage   |
+------------------+

Supervised Process Tree

LemonCore.Application starts these children under a :one_for_one supervisor:

Durable memory moved to lemon_memory, the workspace stores to agent_core, and provider credential-pool rotation to coding_agent (CodingAgent.ProviderPoolRotator); each supervises its own processes.

Browser, media-job, and LSP drivers live in lemon_browser, lemon_media, and lemon_lsp. Core doctor diagnostics may probe them at runtime, but lemon_core must not depend on those apps.

Module Inventory

Configuration

Doctor and Support

media_diagnostics.json in support bundles mirrors the provider-backed media launch lanes with redacted image/TTS/STT/vision/video status, safe reason kinds, proof paths, and target-provider rerun commands. It must not include raw prompts, provider responses, artifact bytes, API keys, or secret names.

usage_diagnostics.json in support bundles mirrors the shared usage summary used by control-plane usage methods: current cost, request and token totals, provider rows, today totals, configured quota limits, and cleanup flags. It must not include raw prompts, responses, channel message bodies, credentials, or secret values.

channel_readiness.json in support bundles mirrors the shared Telegram/Discord launch-gate summary used by channels.status: promoted platform list, gate counts, redacted gate evidence, safe reason kinds, and copy-ready next actions for live proof gates such as Discord slash client-click wait mode. It must not include bot tokens, secret names, chat/channel/guild ids, message bodies, raw proof paths, or raw proof details.

mix lemon.doctor includes usage.status, which reads the same redacted aggregate usage diagnostics and warns when configured run, token, or cost limits are exceeded.

mix lemon.doctor includes channels.readiness, which summarizes the shared Telegram/Discord launch-gate counts from LemonCore.Doctor.ChannelReadiness and points remediation at the first unresolved gate without exposing channel ids, message bodies, proof details, credentials, or secret names.

Secrets

Storage

Memory

Durable memory moved to the lemon_memory app (LemonMemory.*). Core reaches it only through the LemonCore.Store finalize-run hook the runtime configures.

Event System

Routing and Sessions

Operations

Utilities

Quality

Mix Tasks

Provider onboarding preferred_models should favor stable models that actually work on Lemon's live transport path. Do not keep known-broken variants in those default suggestions just because the upstream provider documents them. | mix lemon.onboard.anthropic | Anthropic provider setup | | mix lemon.onboard.antigravity | Antigravity (Google) provider setup with OAuth | | mix lemon.onboard.gemini | Google Gemini CLI / Code Assist setup with OAuth | | mix lemon.onboard.codex | OpenAI Codex provider setup with OAuth and localhost callback capture | | mix lemon.onboard.copilot | GitHub Copilot provider setup with OAuth | | mix lemon.quality | Run all quality checks | | mix lemon.cleanup | Clean up old data | | mix lemon.store.migrate_jsonl_to_sqlite | Migrate JSONL to SQLite | | mix lemon.introspection | Query introspection events | | mix lemon.check_duplicate_tests | Check for duplicate test names |

Configuration System

Config Sources (precedence: env > project > global)

1. Global: ~/.lemon/config.toml
2. Project: <cwd>/.lemon/config.toml
3. Environment: LEMON_* and provider-specific variables override both

Config Sections

• providers -- LLM API keys and base URLs (anthropic, openai, openai-codex, opencode, kimi, google)
• defaults -- Preferred home for default provider/model/thinking level/engine
• runtime -- Runtime behavior (compaction, retry, shell, tools, cli, extensions, theme)
• profiles -- Per-agent profiles with tool policies
• agent -- Legacy alias for runtime/default settings (still supported)
• agents -- Legacy alias for profile settings (still supported)
• tui -- Theme, debug mode
• logging -- File logging, level, rotation
• gateway -- Max concurrent runs, engine bindings, Telegram/Discord/SMS/email settings

Access Patterns

# Cached read (default hot path) - uses ConfigCache when available
config = LemonCore.Config.load(cwd)

# Force reload from disk (updates cache)
config = LemonCore.Config.reload(cwd)

# Access nested values
provider = LemonCore.Config.get(config, [:agent, :default_provider], "anthropic")

# Modular config interface (typed sub-structs, validation)
config = LemonCore.Config.Modular.load(project_dir: cwd)
config = LemonCore.Config.Modular.load!(project_dir: cwd)
{:ok, config} = LemonCore.Config.Modular.load_with_validation(project_dir: cwd)

Hot Reload Flow

LemonCore.ConfigReloader.reload/1
  |-- Acquire reload lock
  |-- Compute file/env/secrets digests
  |-- Compare with cached digests
  |-- Reload .env if changed
  |-- Reload TOML via LemonCore.Config.reload/2
  |-- Compute redacted diff vs previous snapshot
  |-- Update ConfigCache
  |-- Broadcast :config_reloaded on "system" topic
  |-- On failure: keep last good snapshot, emit :config_reload_failed

Environment Variable Overrides

Secrets Management

Storage Model

Secrets are encrypted at rest with AES-256-GCM. Per-secret encryption keys are derived via HKDF-SHA256 from a master key and a random 32-byte salt. The ciphertext, nonce, and salt are base64-encoded and stored in the :secrets_v1 Store table.

Master Key Resolution (in order)

1. macOS Keychain (Lemon Secrets service, default account) -- preferred
2. LEMON_SECRETS_MASTER_KEY environment variable -- fallback
3. ~/.lemon/secrets_master_key file -- local non-keychain fallback
4. Fail with :missing_master_key

The order and the locations are configurable via config :lemon_core, LemonCore.Secrets, key_providers: [:keychain, :env, :file], key_file: ..., env_var: ...; see LemonCore.Secrets.KeyProvider. The key must be base64-encoded 32 bytes.

For local Linux/dev usage, treat ~/.lemon/secrets_master_key as the canonical master key file. ./bin/lemon normalizes LEMON_SECRETS_MASTER_KEY from that file before boot so stale inherited shell env does not override the working local key.

API

# Store a secret
{:ok, metadata} = LemonCore.Secrets.set("api_key", "secret_value", provider: "manual")

# Retrieve a secret
{:ok, value} = LemonCore.Secrets.get("api_key")

# Resolve (store first, then env fallback)
{:ok, value, :store} = LemonCore.Secrets.resolve("api_key")
{:ok, value, :env} = LemonCore.Secrets.resolve("MISSING_FROM_STORE")

# Convenience (returns value or nil)
value = LemonCore.Secrets.fetch_value("ANTHROPIC_API_KEY")

# Check existence
exists? = LemonCore.Secrets.exists?("api_key")

# List (metadata only, no values)
{:ok, metadata_list} = LemonCore.Secrets.list()

# Delete
:ok = LemonCore.Secrets.delete("api_key")

# Status
status = LemonCore.Secrets.status()

Secrets automatically fall back to environment variables of the same name. Use env_fallback: false to disable. Secret reads update usage metadata (usage_count, last_used_at) without mutating updated_at.

Storage Backends

When to Use Each

Configuration

# config/config.exs
config :lemon_core, LemonCore.Store,
  backend: LemonCore.Store.SqliteBackend,
  backend_opts: [path: "/var/lib/lemon/store.sqlite3"]

Wrapper-First Table Access

defmodule MyApp.WidgetStore do
  def put(id, widget), do: LemonCore.Store.put(:widgets, id, widget)
  def claim(id, widget), do: LemonCore.Store.put_new(:widgets, id, widget)
  def get(id), do: LemonCore.Store.get(:widgets, id)
end

MyApp.WidgetStore.put(id, widget)
widget = MyApp.WidgetStore.get(id)

put_new/3 is the insert-if-absent primitive for durable claims. It returns :ok for the first writer and {:error, :exists} for later writers without overwriting the original value.

Store calls are fail-soft: if the GenServer is overloaded/unavailable, write APIs return {:error, :store_unavailable} and read/list APIs return nil/[].

Use the generic table API only inside wrapper modules, backend internals, or explicitly app-local legacy tables. Shared-domain runtime code should go through typed wrappers so table ownership stays localized.

Specialized APIs

Shared-domain callers should prefer typed wrappers:

• Chat state: LemonCore.ChatStateStore.put/2, get/1, delete/1
• Run history: LemonCore.RunStore.append_event/2, finalize/2, history/2, get/1
• Policies: LemonCore.PolicyStore.put_agent/2, put_channel/2, put_session/2, put_runtime/1
• Idempotency: LemonCore.IdempotencyStore.put/3, get/2, delete/2
• Progress mapping: LemonCore.ProgressStore.put/3, get_run/2
• Introspection: LemonCore.IntrospectionStore.append/1, list/1
• Project bindings: LemonCore.ProjectBindingStore.get_override/1, put_override/2, get_dynamic/1
• Exec approvals: LemonCore.ExecApprovalStore.get_pending/1, put_pending/2, policy getters/setters by scope

Agent workspace coordination — goals, kanban boards, and heartbeats — is built on this Store but lives in agent_core as LemonAgent.Workspace.{GoalStore, KanbanStore, HeartbeatStore}.

ReadCache

LemonCore.Store.ReadCache maintains public ETS tables for hot domains (:chat, :runs, :progress, :sessions_index, :telegram_known_targets). Reads bypass the GenServer entirely for O(1) ETS lookup, while writes go through the GenServer which updates both the backend and cache atomically.

Backend Behaviour

Implementing a new backend requires the LemonCore.Store.Backend behaviour:

@callback init(opts()) :: {:ok, state()} | {:error, term()}
@callback put(state(), table(), key(), value()) :: {:ok, state()}
@callback put_new(state(), table(), key(), value()) :: {:ok, state()} | {:exists, state()} | {:error, term()}
@callback get(state(), table(), key()) :: {:ok, value() | nil, state()}
@callback delete(state(), table(), key()) :: {:ok, state()}
@callback list(state(), table()) :: {:ok, [{key(), value()}], state()}

SQLite serializes keys and values with :erlang.term_to_binary/1. The JSONL backend uses a JSON codec that preserves atoms, tuples, structs, and nested map keys for portable append-only files.

Event Bus

Standard Topics

Usage

# Subscribe
LemonCore.Bus.subscribe("run:" <> run_id)

# Receive
receive do
  %LemonCore.Event{type: :delta, payload: payload} -> handle_delta(payload)
  %LemonCore.Event{type: :completed, payload: payload} -> handle_completion(payload)
after
  30_000 -> :timeout
end

# Broadcast
event = LemonCore.Event.new(:run_started, %{engine: "lemon"}, %{run_id: run_id})
LemonCore.Bus.broadcast("session:" <> session_key, event)

# Unsubscribe
LemonCore.Bus.unsubscribe("run:" <> run_id)

Session Keys

Session keys provide stable identifiers for routing and state management.

Formats

• Main: agent:<agent_id>:main[:sub:<sub_id>]
• Channel: agent:<agent_id>:<channel_id>:<account_id>:<peer_kind>:<peer_id>[:thread:<thread_id>][:sub:<sub_id>]

Usage

key = LemonCore.SessionKey.main("my_agent")
# => "agent:my_agent:main"

key = LemonCore.SessionKey.channel_peer(%{
  agent_id: "my_agent", channel_id: "telegram",
  account_id: "bot123", peer_kind: :dm, peer_id: "user456"
})

%{agent_id: id, kind: :main} = LemonCore.SessionKey.parse(key)

Execution Approvals

Tool execution gating with scope-based persistence. Placed in lemon_core so any app can request/resolve approvals without depending on :lemon_router.

Scopes

# Request (blocks until resolved)
case LemonCore.ExecApprovals.request(%{
  run_id: run_id, session_key: session_key,
  tool: "shell", action: %{command: "rm -rf /tmp/old"},
  expires_in_ms: 60_000
}) do
  {:ok, :approved, scope} -> proceed()
  {:ok, :denied} -> halt()
  {:error, :timeout} -> handle_timeout()
end

# Resolve (called by UI/admin)
:ok = LemonCore.ExecApprovals.resolve(approval_id, :approve_session)

Approval requests, resolutions, and timeouts also write redacted approval_* introspection events with approval id, tool, action type/hash, and decision/scope metadata. Raw action payloads, prompts, and secrets are not stored in the timeline. Timeouts also broadcast :approval_resolved with decision: :timeout so operator clients can clear stale pending requests live.

Idempotency

At-most-once deduplication backed by the Store with 24h TTL.

result = LemonCore.Idempotency.execute("messages", msg_id, fn ->
  perform_operation()
end)

Introspection

Canonical event persistence with redaction for tool arguments and sensitive fields.

:ok = LemonCore.Introspection.record(:tool_completed,
  %{tool_name: "exec", result_preview: "ok"},
  run_id: run_id, session_key: session_key, engine: "codex"
)

events = LemonCore.Introspection.list(run_id: run_id, limit: 50)

RouterBridge

Channel adapters forward runs to :lemon_router without compile-time coupling. The router registers itself at startup.

{:ok, run_id} = LemonCore.RouterBridge.submit_run(run_request)
:ok = LemonCore.RouterBridge.handle_inbound(inbound_message)
:ok = LemonCore.RouterBridge.abort_session(session_key, :user_requested)

Returns {:error, :unavailable} when :lemon_router has not registered; callers must handle this gracefully.

Telemetry Events

Dependencies

Everything else is optional — lemon_core boots without it, with the degradation noted below. UUIDs are generated by the vendored LemonCore.UUID.

If you want run history, declare exqlite yourself

exqlite being optional has a consequence that is easy to miss: LemonCore.Application only starts LemonCore.RunHistoryStore when the SQLite NIF is loadable. An application that depends on lemon_core but does not itself declare exqlite boots fine, but run history is disabled — get_run_history/3 returns [] and the finalize-run hooks that persist history do nothing. The only signal is a warning at boot.

Optional dependencies are not inherited: add it to your own deps/0 to enable it.

{:exqlite, "~> 0.34"}

Durable memory has the same requirement in lemon_memory.

Filesystem layout

Everything lemon_core reads or writes by convention hangs off a per-user state directory (~/.lemon) and a per-project one (<project>/.lemon). The defaults are the reference runtime's; LemonCore.Paths makes them configurable so a host application can put its state somewhere not named after Lemon:

config :lemon_core, :paths,
  state_dir: ".lemon",          # per-user and per-project directory name
  config_file: "config.toml",   # file inside it
  home_state_dir: nil,          # pin the per-user directory outright
  global_config: nil            # pin the global config file outright

Renaming :state_dir moves everything derived from it: both config files, the config reloader's watched paths, the default run-history store directory, the doctor's skills and extension directories, and proof artifacts. Two locations are configured separately because they are not part of that layout: the secrets master key (config :lemon_core, LemonCore.Secrets, key_file: ...) and the store backend path (config :lemon_core, LemonCore.Store, backend_opts: [path: ...]).

Testing

# Run all lemon_core tests
mix test apps/lemon_core

# Run specific test file
mix test apps/lemon_core/test/lemon_core/config_test.exs

# Run with coverage
mix test --cover apps/lemon_core

Test Harness

defmodule MyTest do
  use LemonCore.Testing.Case, async: true

  test "example", %{harness: harness, tmp_dir: tmp_dir} do
    path = temp_file!(harness, "config.toml", "[agent]\ndefault_model = \"test\"")
    key = unique_session_key("mytest")
  end
end

Available helpers: unique_token/0, unique_scope/0, unique_session_key/0, unique_run_id/0, temp_file!/3, temp_dir!/2, clear_store_table/1, mock_home!/1, random_master_key/0.

Important Notes

• Never add umbrella app dependencies to lemon_core -- it is the base layer
• Keep module interfaces stable -- other apps depend on them
• LemonCore.Config.load/2 uses the cache by default; LemonCore.Config.reload/2 forces a disk read
• Secret values are never logged or returned by list/status APIs
• SQLite store values serialize with :erlang.term_to_binary/1; JSONL uses a JSON codec with Elixir-term markers for portable persistence
• Events use millisecond timestamps from System.system_time(:millisecond)
• RouterBridge returns {:error, :unavailable} when :lemon_router has not registered
• Dedupe.Ets uses monotonic time for TTL; Idempotency uses wall-clock time
• Config.Modular is the newer typed approach; Config is still the primary interface