Elixir Server Core

Build durable background-processing services in Elixir without the complexity of a full application framework.

Elixir Server Core is a lightweight toolkit for building standalone worker services. It combines HTTP endpoints, supervised job execution, pluggable persistence, retries, scheduling, and observability into a minimal foundation that you can use as a library or fork as a starting point.

Whether you're building a PDF conversion service, media-processing pipeline, webhook receiver, or automation backend, Elixir Server Core provides the essential infrastructure while staying close to OTP principles.

Typical Use Cases

Media processing — video transcoding, image resizing, thumbnail generation
Document workflows — PDF optimization, OCR, format conversion
Webhook ingestion — receive requests and process them asynchronously
Automation services — scheduled jobs, background tasks, integrations
Single-node deployments — SQLite-backed durability on a VPS, edge device, or homelab server
Learning OTP — understand how job queues, workers, supervision trees, and retries are implemented

Why Not Phoenix + Oban?

Phoenix and Oban are excellent tools and should be the default choice for many production systems.

Elixir Server Core targets a different use case: standalone worker services where you don't need a full web application, database layer, or distributed job-processing platform.

If your service only needs:

HTTP endpoints
A durable job queue
Worker supervision
Retries and scheduling
Basic observability

Then a smaller toolkit can be easier to understand, customize, and deploy.

The Problem

Many background-processing services start with a simple requirement:

Accept work, queue it, process it, retry failures, and survive restarts.

Achieving that often means assembling multiple libraries or adopting a larger framework than the problem requires.

The Solution

Elixir Server Core provides the building blocks for specialized worker services:

Plug + Cowboy for HTTP endpoints
OTP supervision trees for fault tolerance
Background job queues with worker pools
Exponential backoff retries
Job scheduling
Telemetry instrumentation
Pluggable persistence

Use in-memory storage for rapid prototyping, SQLite for lightweight durability, or implement the Core.JobStore behaviour to integrate with your preferred database.

The result is a small, understandable foundation that stays close to Elixir's strengths while remaining easy to extend, fork, and deploy.

Features

Forkable server framework for domain-specific services
HTTP server using Plug + Cowboy
OTP supervision trees for fault tolerance
Background job queue with automatic worker execution
In-memory job tracking with full lifecycle management
Worker pool for concurrent job processing
Job scheduling (cron-like future execution)
Exponential backoff retries with configurable max attempts
Observability via Telemetry
Optional Prometheus + Grafana integration (not implemented)
RESTful API with JSON support
Pagination and filtering for job listings
Health check endpoint
Modular and extensible architecture

Quick Start

As a Library (add to deps)

# mix.exs
{:servcore, "~> 0.1"}

# config/config.exs
config :servcore,
  router: MyApp.Router,
  port: 4000,
  job_store: Core.JobStore.SQLite,
  job_store_opts: [database: "priv/jobs.db"]

The framework auto-starts JobQueue, WorkerPool, and Plug.Cowboy with your router.

Manual Supervision (library, full control)

# config/config.exs
config :servcore, start_http: false

# application.ex
children = [
  {Core.Workers.JobQueue, store: Core.JobStore.SQLite, store_opts: [database: "jobs.db"]},
  {Core.Workers.WorkerPool, worker: MyApp.Worker, size: 4},
  {Plug.Cowboy, scheme: :http, plug: MyApp.Router, options: [port: 4000]}
]

As a Fork (customize internals)

Clone, rename the app in mix.exs, edit lib/core/ directly. See FORKING.md.

High-Level Architecture

Client ──HTTP──▶ Router ──▶ OTP Supervision Tree
                                │
                                ├── JobQueue (GenServer)
                                │   ├── Queue: Job IDs
                                │   └── Jobs: Job Data Map
                                │
                                ├── WorkerPool (Supervisor)
                                │   └── Workers (GenServer) × N
                                │       └── Poll & Execute Jobs
                                │
                                └── Telemetry Events
                                    │
                                    ▼
                               /metrics (optional)
                               Prometheus → Grafana

Job Lifecycle

Jobs progress through the following states:

:queued - Job submitted and waiting for a worker
:running - Job claimed by a worker and being processed
:done - Job completed successfully with a result
:failed - Job encountered an error during processing

Jobs can also transition back to :queued when a retry is scheduled after a failure. Each job has a configurable max_attempts (default: 3) and uses exponential backoff between retries.

Jobs remain in the queue throughout their lifecycle, allowing you to track their complete history and status via the API. The worker pool polls the queue, claims the next available job, executes it, and updates its status accordingly.

Project Structure

elixir_server_core/
├── lib/
│   ├── core/
│   │   ├── http/
│   │   │   ├── router.ex              # HTTP routing and endpoints
│   │   │   └── base_router.ex         # Base router for forking
│   │   ├── workers/
│   │   │   ├── job.ex                 # Job struct definition
│   │   │   ├── job_queue.ex           # Job queue GenServer
│   │   │   ├── worker.ex              # Background job worker
│   │   │   └── worker_pool.ex         # Worker pool supervisor
│   │   ├── telemetry/
│   │   │   ├── events.ex              # Telemetry event definitions
│   │   │   └── metrics.ex             # Telemetry metrics definitions
│   │   └── capability/                # Optional reusable capabilities
│   │       ├── http.ex                # Alternative HTTP capability
│   │       ├── work_queue.ex          # Work queue capability
│   │       ├── metrics.ex             # Capability metrics
│   │       └── server_template.ex     # Template for forked servers
│   └── elixir_server_core/
│       └── application.ex             # Main application supervisor
├── config/
│   └── config.exs
├── test/
│   ├── elixir_server_core_test.exs   # Integration tests
│   └── test_helper.exs
├── mix.exs                            # Project dependencies
├── mix.lock
└── README.md

Getting Started

Requirements

Elixir 1.14 or newer
Erlang/OTP 26 or newer

Setup

# Clone the repository
git clone <repository-url>
cd servcore

# Install dependencies
mix deps.get

# Compile the project
mix compile

Running the Server

mix run --no-halt

Default address:

http://localhost:4000

You should see:

[info] Starting server on port 4000
[info] http://localhost:4000
[info] Worker started

API Endpoints

Overview

Method	Endpoint	Description
GET	`/`	Root endpoint - server status
GET	`/health`	Health check
GET	`/stats`	Job statistics
POST	`/jobs`	Submit a new job
POST	`/jobs/schedule`	Schedule a job for future execution
GET	`/jobs`	List all jobs
GET	`/jobs/:id`	Get a specific job by ID

Endpoint Details

`GET /` - Root Endpoint

Returns a simple status message.

Request:

curl http://localhost:4000/

Response:

Server is running

`GET /health` - Health Check

Returns the health status of the server.

Request:

curl http://localhost:4000/health

Response:

{"status": "OK"}

If the JobQueue process is not running, returns:

{"status": "DEGRADED"}

`GET /stats` - Job Statistics

Returns aggregate counts of jobs by status.

Request:

curl http://localhost:4000/stats

Response:

{
  "queued": 2,
  "running": 1,
  "done": 5,
  "failed": 0,
  "total": 8
}

`POST /jobs` - Submit a New Job

Submits a new job to the queue for processing.

Request:

curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "process_data", "value": 42}}'

Response (202 Accepted):

{
  "message": "Job accepted",
  "job_id": 123
}

Error Response (400 Bad Request):

{
  "error": "Missing 'payload' field"
}

Optional Parameters:

Parameter	Type	Description
`max_attempts`	integer	Maximum retry attempts (default: 3)

Examples:

# Simple task
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "send_email", "recipient": "user@example.com"}}'

# Complex payload
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "generate_report", "filters": {"date_range": "2024-01-01:2024-12-31", "type": "sales"}}}'

# Batch processing
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "process_batch", "items": [1, 2, 3, 4, 5]}}'

# With custom retry limit
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "critical_task"}, "max_attempts": 5}'

`POST /jobs/schedule` - Schedule a Job

Schedules a job to run at a specific future time.

Request:

curl -X POST http://localhost:4000/jobs/schedule \
  -H "Content-Type: application/json" \
  -d '{
    "payload": {"task": "send_reminder"},
    "run_at": "2025-12-31T23:59:59Z"
  }'

Response (202 Accepted):

{
  "message": "Job scheduled",
  "job_id": 456,
  "run_at": "2025-12-31T23:59:59Z"
}

Error Response (400 Bad Request):

{
  "error": "Required fields: payload (object), run_at (ISO8601)"
}

`GET /jobs` - List All Jobs

Returns jobs in the queue with their current status. Supports filtering by status and pagination. Sorted by insertion time descending.

Query Parameters:

Parameter	Type	Description
`status`	string	Filter by status: `queued`, `running`, `done`, `failed`
`page`	integer	Page number (default: 1)
`per_page`	integer	Items per page, max 200 (default: 50)

Request:

curl http://localhost:4000/jobs

Response (200 OK):

[
  {
    "id": 123,
    "payload": {"task": "process_data", "value": 42},
    "status": "done",
    "inserted_at": "2025-12-28T17:24:48.957749Z",
    "started_at": "2025-12-28T17:24:49.566352Z",
    "finished_at": "2025-12-28T17:24:49.667314Z",
    "result": {
      "status": "completed",
      "job_id": 123,
      "processed_at": "2025-12-28T17:24:49.667198Z"
    }
  },
  {
    "id": 124,
    "payload": {"task": "send_email"},
    "status": "running",
    "inserted_at": "2025-12-28T17:25:01.123456Z",
    "started_at": "2025-12-28T17:25:02.234567Z",
    "finished_at": null,
    "result": null
  },
  {
    "id": 125,
    "payload": {"task": "generate_report"},
    "status": "queued",
    "inserted_at": "2025-12-28T17:25:05.345678Z",
    "started_at": null,
    "finished_at": null,
    "result": null
  }
]

Pretty Print Response:

curl http://localhost:4000/jobs | jq

Filter by Status (API query parameter):

# Show only completed jobs
curl "http://localhost:4000/jobs?status=done"

# Show only running jobs
curl "http://localhost:4000/jobs?status=running"

Pagination:

# Get page 2 with 10 items per page
curl "http://localhost:4000/jobs?page=2&per_page=10"

Filter by Status (using jq):

# Count jobs by status
curl -s http://localhost:4000/jobs | jq 'group_by(.status) | map({status: .[0].status, count: length})'

`GET /jobs/:id` - Get Specific Job

Returns detailed information about a specific job.

Request:

curl http://localhost:4000/jobs/123

Response (200 OK):

{
  "id": 123,
  "payload": {"task": "process_data", "value": 42},
  "status": "done",
  "inserted_at": "2025-12-28T17:24:48.957749Z",
  "started_at": "2025-12-28T17:24:49.566352Z",
  "finished_at": "2025-12-28T17:24:49.667314Z",
  "result": {
    "status": "completed",
    "job_id": 123,
    "processed_at": "2025-12-28T17:24:49.667198Z"
  }
}

Error Response (404 Not Found):

{
  "error": "Job not found"
}

Examples:

# Get job details
curl http://localhost:4000/jobs/123

# Pretty print with jq
curl http://localhost:4000/jobs/123 | jq

# Extract specific fields
curl -s http://localhost:4000/jobs/123 | jq '{id: .id, status: .status, result: .result}'

# Check if job is complete
curl -s http://localhost:4000/jobs/123 | jq '.status == "done"'

Complete Workflow Example

1. Submit Multiple Jobs

# Submit job 1
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "backup_database"}}'

# Submit job 2
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "send_notifications"}}'

# Submit job 3 with custom retry limit
curl -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "generate_reports"}, "max_attempts": 5}'

# Schedule a job for tomorrow
RUN_AT=$(date -u -d '+1 day' +%Y-%m-%dT%H:%M:%SZ)
curl -X POST http://localhost:4000/jobs/schedule \
  -H "Content-Type: application/json" \
  -d "{\"payload\": {\"task\": \"daily_cleanup\"}, \"run_at\": \"$RUN_AT\"}"

2. Monitor Job Progress

# List all jobs
curl http://localhost:4000/jobs | jq

# Quick stats overview
curl http://localhost:4000/stats | jq

# Watch jobs in real-time (refresh every 2 seconds)
watch -n 2 'curl -s http://localhost:4000/jobs | jq'

3. Check Specific Job Status

# Get job by ID (replace with actual job ID)
curl http://localhost:4000/jobs/1 | jq

# Poll until job is done
while true; do
  STATUS=$(curl -s http://localhost:4000/jobs/1 | jq -r '.status')
  echo "Job status: $STATUS"
  if [ "$STATUS" = "done" ] || [ "$STATUS" = "failed" ]; then
    break
  fi
  sleep 1
done

4. Analyze Results

# Get all completed jobs with their results
curl -s http://localhost:4000/jobs | jq '[.[] | select(.status == "done") | {id: .id, task: .payload.task, result: .result}]'

# Calculate average processing time
curl -s http://localhost:4000/jobs | jq '[.[] | select(.started_at != null and .finished_at != null)] | map((.finished_at | fromdateiso8601) - (.started_at | fromdateiso8601)) | add / length'

Testing

Run Tests

# Run all tests
mix test

# Run tests with coverage
mix test --cover

# Run specific test file
mix test test/elixir_server_core_test.exs

# Run tests in watch mode (requires mix_test_watch)
mix test.watch

Manual Testing Script

Create a file test_api.sh:

#!/bin/bash

echo "=== Testing Elixir Server Core API ==="
echo

echo "1. Health Check"
curl -s http://localhost:4000/health | jq
echo

echo "2. Stats"
curl -s http://localhost:4000/stats | jq
echo

echo "3. Submit Job 1"
JOB1=$(curl -s -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "test_job_1"}}')
echo $JOB1 | jq
JOB1_ID=$(echo $JOB1 | jq -r '.job_id')
echo

echo "4. Submit Job 2"
JOB2=$(curl -s -X POST http://localhost:4000/jobs \
  -H "Content-Type: application/json" \
  -d '{"payload": {"task": "test_job_2"}, "max_attempts": 5}')
echo $JOB2 | jq
JOB2_ID=$(echo $JOB2 | jq -r '.job_id')
echo

echo "5. Schedule Future Job"
RUN_AT=$(date -u -d '+1 hour' +%Y-%m-%dT%H:%M:%SZ)
curl -s -X POST http://localhost:4000/jobs/schedule \
  -H "Content-Type: application/json" \
  -d "{\"payload\": {\"task\": \"future_job\"}, \"run_at\": \"$RUN_AT\"}" | jq
echo

echo "6. Wait for processing..."
sleep 2
echo

echo "7. Get All Jobs"
curl -s http://localhost:4000/jobs | jq
echo

echo "8. Get Job 1 Details"
curl -s http://localhost:4000/jobs/$JOB1_ID | jq
echo

echo "9. Get Job 2 Details"
curl -s http://localhost:4000/jobs/$JOB2_ID | jq
echo

echo "=== Test Complete ==="

Make it executable and run:

chmod +x test_api.sh
./test_api.sh

Forking the Server

You can fork this server to create domain-specific applications. Here's an example:

Creating a Music Server

defmodule MyMusicServer.Application do
  use Application

  def start(_type, _args) do
    children = [
      # Core capabilities
      Core.Workers.JobQueue,
      Core.Workers.Worker,
      
      # Custom HTTP router with music-specific endpoints
      {Plug.Cowboy, 
        scheme: :http, 
        plug: MyMusicServer.Router, 
        options: [port: 5000]
      },
      
      # Add your domain-specific services
      MyMusicServer.Library,
      MyMusicServer.Player,
      MyMusicServer.Playlist
    ]

    opts = [strategy: :one_for_one, name: MyMusicServer.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Extending Worker Behavior

Override the perform_work/1 function to add custom job handling:

defmodule MyMusicServer.Worker do
  use GenServer
  alias Core.Workers.JobQueue

  # ... (same setup as Core.Workers.Worker)

  defp perform_work(job) do
    case job.payload do
      %{"task" => "transcode_audio", "file" => file} ->
        transcode_audio(file)
      
      %{"task" => "generate_waveform", "track_id" => id} ->
        generate_waveform(id)
      
      %{"task" => "sync_library"} ->
        sync_library()
      
      _ ->
        %{error: "Unknown task type"}
    end
  end

  defp transcode_audio(file) do
    # Custom audio processing logic
    %{status: "transcoded", output: "#{file}.mp3"}
  end
  
  # ... more custom handlers
end

Architecture Decisions

Why GenServer for Job Queue?

Serialized Access: Ensures thread-safe operations on the queue
State Management: Natural fit for maintaining queue and job state
Supervision: Automatic restart on crashes
Telemetry Integration: Built-in metrics and monitoring
Retry Logic: Centralized handling of exponential backoff and re-enqueueing

Why Keep Jobs in Queue?

Full History: All jobs remain queryable after completion
Simpler Design: No need for separate storage (ETS, DB)
Atomic Updates: GenServer calls ensure consistency
Debugging: Easy to inspect entire job lifecycle

Job Storage Structure

%{
  queue: :queue.new(),     # Queue of job IDs (FIFO)
  jobs: %{                 # Map of job ID to Job struct
    123 => %Job{...},
    124 => %Job{...}
  }
}

This dual structure allows:

Fast FIFO queue operations
O(1) job lookup by ID
In-place status updates
Full job history retention

Performance Considerations

Current Limitations

In-Memory Only: Jobs are lost on server restart
Polling Overhead: Workers poll every second
No Job Priorities: All jobs are processed FIFO

Scaling Strategies

For production deployments, consider:

Persistent Storage: Add PostgreSQL or Redis for job persistence
Job Cleanup: Archive completed jobs after N days
Priority Queue: Implement job prioritization
Distributed Queue: Use RabbitMQ or Kafka for distributed systems

Configuration

Port Configuration

Edit lib/elixir_server_core/application.ex:

port = System.get_env("PORT", "4000") |> String.to_integer()

Then run:

PORT=8080 mix run --no-halt

Worker Poll Interval

Edit lib/core/workers/worker.ex:

@poll_interval 500  # Poll every 500ms instead of 1000ms

Observability

Logging

The server logs key events:

[info] Starting server on port 4000
[info] Worker started
[info] Executing job 123
[info] Job 123 completed successfully
[error] Job 124 failed: %ArgumentError{message: "invalid data"}

Telemetry Events

The following telemetry events are emitted:

[:server, :http, :start] - HTTP request started
[:server, :http, :stop] - HTTP request completed
[:core, :job, :start] - Job execution started
[:core, :job, :stop] - Job execution completed
[:core, :job, :error] - Job execution failed

Adding Prometheus Integration

To expose metrics, add to your supervision tree:

children = [
  # ... existing children
  {TelemetryMetricsPrometheus, 
    metrics: Core.Capability.Metrics.metrics()
  }
]

Then access metrics at http://localhost:9568/metrics

Configuration Reference

Option	Type	Default	Description
`:router`	module	`Core.HTTP.Router`	Plug router module
`:port`	integer	`4000` (or `PORT` env)	HTTP server port
`:ip`	tuple	`{0,0,0,0}`	Bind address
`:worker`	module	`Core.Workers.Worker`	Worker module for processing jobs
`:worker_pool_size`	integer	CPU cores	Number of concurrent workers
`:job_store`	module	`Core.JobStore.Memory`	Persistence backend
`:job_store_opts`	keyword	`[]`	Options passed to the store
`:start_http`	boolean	`true`	Start `Plug.Cowboy` automatically
`:start_workers`	boolean	`true`	Start `WorkerPool` automatically

Set start_http: false when integrating into an existing Phoenix application or when you want full control over the HTTP server.

Storage Backend Notes

SQLite throughput ceiling: The built-in SQLite adapter opens a new connection for every operation. This is simple and stateless, but it caps throughput at roughly ~1,000 operations per second on a typical SSD. If you need higher throughput, implement a stateful Core.JobStore adapter that uses a connection pool (e.g., DBConnection with Postgrex) or keeps a single long-lived connection.

Troubleshooting

Server won't start

# Check if port is already in use
lsof -i :4000

# Kill existing process
kill -9 <PID>

# Or use a different port
PORT=4001 mix run --no-halt

Jobs not processing

# Check if worker is running
curl http://localhost:4000/health

# View logs for errors
mix run --no-halt

# Verify job was submitted
curl http://localhost:4000/jobs | jq

JSON encoding errors

Ensure all structs used in responses have @derive Jason.Encoder:

defmodule MyStruct do
  @derive Jason.Encoder
  defstruct [:field1, :field2]
end

Open Source and Contributions

This project is fully open source under the MIT License. Contributions are welcome in the form of:

Adding metrics and instrumentation
Building Prometheus + Grafana integration
Implementing domain-specific servers (music, PDF, etc.)
Adding persistent storage backends
Improving documentation and tests
Performance optimizations
Security enhancements

Contributing Guidelines

Fork the repository
Create a feature branch: git checkout -b feature/my-feature
Make your changes with tests
Run tests: mix test
Commit: git commit -am 'Add my feature'
Push: git push origin feature/my-feature
Open a Pull Request

License

MIT License - see LICENSE file for details

Maintainer

DarynOngera

For questions, issues, or feature requests, please open an issue on GitHub.

Resources

Roadmap

Completed:

Worker pool for parallel processing
Job scheduling (cron-like)
Job retries with exponential backoff
SQLite persistence backend
Pluggable Core.JobStore behaviour for custom databases

Planned:

PostgreSQL persistence backend (via Core.JobStore.SQL + Postgrex)
Job priorities
Prometheus + Grafana integration
Authentication and authorization
Admin dashboard UI
Docker and Kubernetes deployment guides
Performance benchmarking suite

Elixir Server Core

Typical Use Cases

Why Not Phoenix + Oban?

The Problem

The Solution

Features

Quick Start

As a Library (add to deps)

Manual Supervision (library, full control)

As a Fork (customize internals)

High-Level Architecture

Job Lifecycle

Project Structure

Getting Started

Requirements

Setup

Running the Server

API Endpoints

Overview

Endpoint Details

GET / - Root Endpoint

GET /health - Health Check

GET /stats - Job Statistics

POST /jobs - Submit a New Job

POST /jobs/schedule - Schedule a Job

GET /jobs - List All Jobs

GET /jobs/:id - Get Specific Job

Complete Workflow Example

1. Submit Multiple Jobs

2. Monitor Job Progress

3. Check Specific Job Status

4. Analyze Results

Testing

Run Tests

Manual Testing Script

Forking the Server

Creating a Music Server

Extending Worker Behavior

Architecture Decisions

Why GenServer for Job Queue?

Why Keep Jobs in Queue?

Job Storage Structure

Performance Considerations

Current Limitations

Scaling Strategies

Configuration

Port Configuration

Worker Poll Interval

Observability

Logging

Telemetry Events

Adding Prometheus Integration

Configuration Reference

Storage Backend Notes

Troubleshooting

Server won't start

Jobs not processing

JSON encoding errors

Open Source and Contributions

Contributing Guidelines

License

Maintainer

Resources

Roadmap

`GET /` - Root Endpoint

`GET /health` - Health Check

`GET /stats` - Job Statistics

`POST /jobs` - Submit a New Job

`POST /jobs/schedule` - Schedule a Job

`GET /jobs` - List All Jobs

`GET /jobs/:id` - Get Specific Job