mix fly_deploy.hot

Hot code upgrades for Elixir applications running on Fly.io without restarts or downtime.

FlyDeploy enables zero-downtime deployments by upgrading running BEAM processes with new code while preserving their state. Unlike traditional deployments that restart your entire application, hot upgrades suspend processes, swaps in new code, migrates state via code_change callbacks, then resumes processes for seamless code upgrades.

Features

Comparison with OTP releases and release handlers

FlyDeploy provides a simplified approach to hot code upgrades compared to traditional OTP releases and the release_handler module. Understanding the differences will help you choose the best tool for your needs.

Traditional OTP release upgrades

OTP's release_handler provides the canonical hot upgrade mechanism for Erlang/Elixir applications:

FlyDeploy's simplified approach

FlyDeploy takes a different approach optimized for containerized deployments and simplified upgrades, where we accept that some changes require cold deploys:

Key Differences

State Management:

Metadata Storage:

Build Artifacts:

Limitations

Compared to OTP's release_handler, FlyDeploy cannot:

When to Use Each

Use FlyDeploy when:

Use OTP release_handler when:

Installation

Add fly_deploy to your list of dependencies in mix.exs:

def deps do
  [
    {:fly_deploy, "~> 0.1.0"}
  ]
end

Quick Start

1. Configure Fly Secrets

Set up AWS credentials for storage:

fly storage create -a myapp -n my-releases-bucket

Or for existing creds:

fly secrets set AWS_ACCESS_KEY_ID=<key> AWS_SECRET_ACCESS_KEY=<secret>

You will also need to set a secret on the app of the Fly API token for the orchestrator machine:

fly secrets set FLY_API_TOKEN=$(fly tokens create machine-exec)

2. Add Startup Hook

In your Application.start/2, you must call startup_reapply_current/1before starting your supervision tree. This will reapply any previously applied hot upgrade on top of the running container image, allowing hot deploys to survive machine restarts.

defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    # Apply any hot upgrade that builds on top of our static container image on startup
    FlyDeploy.startup_reapply_current(:my_app)

    children = [
      # your supervision tree
    ]

    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

3. Deploy

Run a hot deployment:

mix fly_deploy.hot

That's it. Your application will upgrade to the new code without restarting.

How It Works

Hot Upgrade Process

When you run mix fly_deploy.hot:

  1. Build Phase - Creates a new Docker image with fly deploy --build-only
  2. Orchestrator Phase - Spawns a temporary machine with the new image
  3. Tarball Phase - Orchestrator extracts all .beam files and creates a tarball
  4. Upload Phase - Uploads tarball to S3 storage
  5. Metadata Phase - Updates deployment metadata with hot upgrade information
  6. Reload Phase - Each running machine downloads the tarball
  7. Extract Phase - Beam files are extracted and copied to disk
  8. Detection Phase - :code.modified_modules() identifies changed modules
  9. Suspension Phase - Processes using changed modules are suspended with :sys.suspend/1
  10. Code Load Phase - Old modules are purged and new versions loaded
  11. Migration Phase - :sys.change_code/4 is called on each process
  12. Resume Phase - Processes are resumed with :sys.resume/1

Total suspension time is typically under 1 second.

Startup Reapply

When a machine restarts after a hot upgrade (due to crashes, scaling, or restarts):

  1. FlyDeploy.startup_reapply_current/1 checks for current hot upgrades
  2. Compares the machine's Docker image ref with stored metadata
  3. If refs match and a hot upgrade exists, downloads and applies it
  4. Uses :c.lm() to load all modified modules before supervision tree starts
  5. No process suspension needed since supervision tree hasn't started yet

This ensures machines that restart remain consistent with machines that were hot-upgraded.

Configuration

Environment Variables

Required:

Optional:

fly.toml Configuration

Environment variables from your [env] section are automatically passed to the orchestrator:

[env]
  AWS_ENDPOINT_URL_S3 = "https://t3.storage.dev"
  AWS_REGION = "auto"

Mix Configuration

In config/config.exs:

config :fly_deploy,
  bucket: "my-custom-bucket",  # Optional - defaults to BUCKET_NAME env var
  max_concurrency: 10,         # Max concurrent machine upgrades (default: 20)
  suspend_timeout: 15_000,     # Timeout for suspending each process in ms (default: 10_000)
  env: %{
    "CUSTOM_VAR" => "value"
  }

Bucket Configuration: FlyDeploy looks up the S3 bucket name from:

  1. Mix config (:bucket key above) - if explicitly configured
  2. BUCKET_NAME environment variable - automatically set by fly storage create

If neither is configured, deployment will fail. When you run fly storage create, the BUCKET_NAME env var is automatically set on all your machines, so no additional configuration is needed.

CLI Options

The mix fly_deploy.hot task supports several options:

Examples

Basic hot deployment:

mix fly_deploy.hot

Use staging configuration:

mix fly_deploy.hot --config fly-staging.toml

Preview changes without executing:

mix fly_deploy.hot --dry-run

Use pre-built image:

mix fly_deploy.hot --skip-build --image registry.fly.io/my-app:deployment-123

Safety and Error Handling

FlyDeploy uses a 4-phase upgrade cycle to safely upgrade running processes:

  1. Phase 1: Suspend all changed processes - All affected processes are suspended with :sys.suspend/1 before any code loading
  2. Phase 2: Load all changed code - New code is loaded globally using :code.purge/1 and :code.load_file/1 while processes are safely suspended
  3. Phase 3: Upgrade all processes - Each suspended process has :sys.change_code/4 called to trigger its code_change/3 callback
  4. Phase 4: Resume all processes - All processes are resumed with :sys.resume/1
  5. Phase 5: Trigger LiveView reloads (if applicable) - Phoenix LiveView pages automatically re-render with new code

This 4-phase approach eliminates race conditions where one upgraded process calls another that still has old code.

Phoenix LiveView Integration

If you're using Phoenix LiveView, FlyDeploy automatically triggers re-renders after hot upgrades:

CSS Hot Reload

When static assets like CSS change during a hot upgrade, users would normally need to hard refresh to see the new styles. The hot_reload_css component automatically reloads stylesheets when the static manifest changes.

Add to your app layout (e.g., app.html.heex):

<FlyDeploy.Components.hot_reload_css socket={@socket} />

For multiple stylesheets, specify the asset name:

<FlyDeploy.Components.hot_reload_css socket={@socket} asset="app.css" />
<FlyDeploy.Components.hot_reload_css socket={@socket} asset="admin.css" />

Requirements:

The component uses a colocated JavaScript hook that detects when the static manifest version changes and automatically updates stylesheet hrefs, preserving any CDN/static host configuration.

Rollback Strategy

Hot upgrades are forward-only. Once new code is loaded into the BEAM VM, FlyDeploy cannot roll it back. If a hot upgrade causes issues, perform a cold deploy to a known good version:

fly deploy

The cold deploy will replace both the base Docker image and any hot upgrade state. This is similar to how OTP release upgrades work - they are also forward-only unless you build explicit downgrade instructions.

Storage Structure

FlyDeploy stores two types of objects in S3:

Release Tarballs

Path: releases/<app>-<version>.tar.gz

Contains all .beam files from /app/lib/**/ebin/*.beam with relative paths like:

lib/my_app-1.2.3/ebin/Elixir.MyModule.beam
lib/my_app-1.2.3/ebin/Elixir.MyModule.Server.beam

Deployment Metadata

Path: releases/<app>-current.json

Tracks current deployment state:

{
  "image_ref": "registry.fly.io/my-app:deployment-01K93Q...",
  "hot_upgrade": {
    "version": "1.2.3",
    "source_image_ref": "registry.fly.io/my-app:deployment-01K94R...",
    "tarball_url": "https://t3.storage.dev/bucket/releases/my_app-1.2.3.tar.gz",
    "deployed_at": "2024-01-15T10:30:00Z"
  }
}

When to use cold deploy instead of hot upgrade

Testing

Run E2E tests (requires a deployed Fly app):

mix test