PhoenixKitSync

Peer-to-peer data sync module for PhoenixKit.

Provides bidirectional data synchronization between PhoenixKit instances — sync between dev and prod, dev and dev, or different websites entirely.

Installation

Add to your parent app's mix.exs:

{:phoenix_kit_sync, "~> 0.2"}

The module is auto-discovered via PhoenixKit's beam scanning — no additional configuration needed. Enable it from the admin dashboard under Modules.

Architecture

Automatic Cross-Site Registration

The module uses permanent connections with automatic cross-site registration:

1. Sender creates a connection pointing to a remote site's URL
2. System automatically notifies the remote site via API
3. Remote site registers the connection based on their incoming settings:
   • Auto Accept: Connection activates immediately
   • Require Approval: Connection appears as pending
   • Require Password: Only accepts with correct password
   • Deny All: Rejects the connection request
4. Both sites have matching connection records for data sync
5. All transfers are tracked in the history with full audit trail

Connection Types

• Sender: "I allow this remote site to pull data from me"
• Receiver: "I can pull data from this remote site" (auto-created via API)

When you create a sender connection, the remote site automatically receives a corresponding receiver connection.

Features

• Ephemeral Code-Based Transfers: One-time manual sync with 8-character secure codes
• Permanent Token-Based Connections: Recurring sync with full access controls
• Token-Based Authentication: Secure tokens, hashed in database, shown only once
• Approval Modes: Auto-approve, require approval, or per-table approval
• Access Controls: Allowed/excluded tables, download limits, record limits
• Security Features: IP whitelist, time-of-day restrictions, expiration dates
• Conflict Resolution: skip, overwrite, merge, append strategies
• Transfer Tracking: Full history with statistics and approval workflow
• Audit Trail: Track who created, approved, suspended, or revoked connections
• Real-Time Progress: Live tracking of sync operations
• Background Import: Async processing via Oban workers
• Cross-Site Protocol: HTTP API + WebSocket for data transfer
• Activity Audit: Every connection mutation (create, approve, suspend, revoke, reactivate, delete, token rotate) and every transfer mutation (create, complete, fail, cancel, approve, deny) writes a sync.<resource>.<verb> row to phoenix_kit_activities with the actor's UUID — visible in the core admin Activity feed
• Centralised error translation: Context functions return {:error, :atom} tuples; UI / API layers translate via PhoenixKitSync.Errors.message/1, which dispatches to gettext-wrapped strings (translations live in core phoenix_kit)

Connection Settings

Sender-Side Controls

Connection Statuses

Incoming Connection Settings

Control how your site handles connection requests from other sites:

Workflow

Setting Up a Sender Connection

1. Navigate to the Sync connections page in the admin dashboard
2. Click "New Connection"
3. Enter a name and the remote site's URL
4. Configure access controls (approval mode, tables, limits)
5. Save — the connection is created and token generated
6. The remote site is notified automatically!
   • If successful, the connection appears in their list
   • Based on their settings, it may be auto-approved or pending
7. If notification fails, share the token manually as a fallback

What Happens on the Remote Site

When you create a sender connection:

• Your site calls POST {remote_url}/{prefix}/sync/api/register-connection
• The remote site creates a matching receiver connection
• Based on their incoming mode:
  • Auto Accept: Ready to use immediately
  • Require Approval: Admin must approve in their connections list
  • Require Password: You need to provide their password
  • Deny All: Connection is rejected

Choosing Tables to Pull

• Primary keys decide how a table is imported. With a single-column key, rows are matched against local ones by key, then by unique columns. A row matched by unique columns under another key is remapped, so tables that reference it point at the local row (uuid, integer and text keys). A composite key such as (lang, value) is imported as is: conflicts resolve on the whole key. A table with no primary key is not pulled at all — the pull reports it as an error, because without a key every repeat pull would insert the same rows again.
• Conflict strategy. skip leaves a matched row alone; overwrite writes the sender's values onto it; merge does the same but keeps the local value wherever the sender's is NULL. On the bulk pull, append inserts unmatched rows under a new key and leaves matched rows alone. Unique columns other than the key: on the bulk pull, a row with a new single-column key whose unique columns match a local row is that row (remapped, then left alone or updated per the strategy). A row that would clash with another local row's unique value (a composite key, the precise pull, or an update of an existing key) is skipped under skip, overwrite and merge.
• The table must exist on this site. A table missing here is reported as an error rather than pulled; create it first (Precise Transfer > Create Table). Both checks run before anything is requested from the sender, so they do not use up its download or record limits.
• Leave out tables that a database trigger maintains. Some tables are projections that a trigger keeps in step with another table (for example a slug lookup table filled by an AFTER INSERT OR UPDATE trigger on the table that owns the slugs). Pull the source table only: importing it runs the trigger on this site and fills the projection here. Pulling the projection as well can add rows the source does not have, and the trigger may then reject a later write to the source. Sync cannot tell such tables apart; leaving them out is the admin's call.
• A table that fails as a whole (sender offline, table missing here, no primary key, an import that raised) does not stop the run: its row on the results screen shows the reason and the run ends as "completed with errors".

Programmatic API

Connection Management

alias PhoenixKitSync.Connections

# Create a sender connection. The raw token is returned in the third
# tuple element — it's only available at creation time; afterwards only
# the SHA-256 hash is stored.
{:ok, connection, token} =
  Connections.create_connection(%{
    "name" => "Production Backup",
    "direction" => "sender",
    "site_url" => "https://backup.example.com",
    "approval_mode" => "auto_approve",
    "allowed_tables" => ["users", "posts"],
    "max_downloads" => 100,
    "created_by_uuid" => current_user.uuid
  })

# Approve a pending connection (admin_user_uuid is recorded as the actor
# on the resulting `sync.connection.approved` activity-log row).
{:ok, connection} = Connections.approve_connection(connection, admin_user_uuid)

# Suspend a connection
{:ok, connection} = Connections.suspend_connection(connection, admin_user_uuid, "Security audit")

# Reactivate a suspended connection. Pass `actor_uuid:` in opts so the
# activity log records who reactivated it.
{:ok, connection} = Connections.reactivate_connection(connection, actor_uuid: admin_user_uuid)

# Revoke permanently
{:ok, connection} = Connections.revoke_connection(connection, admin_user_uuid, "No longer needed")

# Rotate the auth token (returns the new raw token, hash stored in DB).
# Logged as `sync.connection.token_regenerated`.
{:ok, connection, new_token} = Connections.regenerate_token(connection, actor_uuid: admin_user_uuid)

# Validate a token (used by receiver when connecting)
case Connections.validate_connection(token, client_ip) do
  {:ok, connection} -> # Token is valid, connection is active
  {:error, :invalid_token} -> # Token doesn't exist
  {:error, :connection_expired} -> # Expired or revoked
  {:error, :download_limit_reached} -> # Max downloads reached
  {:error, :ip_not_allowed} -> # IP not in whitelist
  {:error, :outside_allowed_hours} -> # Outside time window
end

Transfer Tracking

alias PhoenixKitSync.Transfers

# Record a transfer
{:ok, transfer} = Transfers.create_transfer(%{
  direction: "send",
  connection_uuid: connection.uuid,
  table_name: "users",
  records_transferred: 150,
  bytes_transferred: 45000,
  status: "completed"
})

# Get transfer history
transfers = Transfers.list_transfers(
  connection_uuid: connection.uuid,
  direction: "send",
  status: "completed"
)

# Get statistics for a connection
stats = Transfers.connection_stats(connection.uuid)
# => %{total_transfers: 25, total_records: 5000, total_bytes: 1500000}

System Control

# Enable/disable
PhoenixKitSync.enabled?()
PhoenixKitSync.enable_system()
PhoenixKitSync.disable_system()
PhoenixKitSync.get_config()

# Local database inspection
{:ok, tables} = PhoenixKitSync.list_tables()
{:ok, schema} = PhoenixKitSync.get_schema("users")
{:ok, records} = PhoenixKitSync.export_records("users", limit: 100)

# Import with conflict strategy
{:ok, result} = PhoenixKitSync.import_records("users", records, :skip)

Remote Client

alias PhoenixKitSync.Client

{:ok, client} = Client.connect("https://sender.com", "ABC12345")
{:ok, tables} = Client.list_tables(client)
{:ok, result} = Client.transfer(client, "users", strategy: :skip)
Client.disconnect(client)

API Endpoints

Cross-site communication endpoints (under the configured URL prefix):

• POST /sync/api/register-connection — Register incoming connection
• POST /sync/api/delete-connection — Delete a connection
• POST /sync/api/verify-connection — Verify connection token
• POST /sync/api/update-status — Update connection status
• POST /sync/api/get-connection-status — Query connection status
• POST /sync/api/list-tables — List available tables
• POST /sync/api/pull-data — Pull table data. Optional record filter: ids (up to 1000 key values) or id_start / id_end (an integer range; either bound may be left out), on a table with a single-column key. A sender that applied the filter adds "filtered": true to the answer; a receiver that sent a filter and gets an answer without it (a sender from before filters) imports nothing and reports it. Such a sender still counts the whole table in its downloads_used / records_downloaded and records the send as completed, although nothing was kept. The filter is compared in the key's own type (integer, uuid or text keys; a range needs an integer key); a filter the sender cannot apply is a 400 whose error_code says why (invalid_filter, unsupported_key_type, filter_needs_single_key). table-records takes the same filter. This narrows what table-records accepted for previews: a filter on a table with a composite key or none (a view included), an empty ids list, or ids together with a range is now a 400, where it used to filter on the first key column or on id.
• POST /sync/api/table-schema — Get table schema
• POST /sync/api/table-records — Get table records
• GET /sync/api/status — Check module status

When notifying a remote site, ConnectionNotifier prepends the remote's URL prefix to these paths. By default it mirrors the local site's PhoenixKit.Config.get_url_prefix/0, which is correct when both deployments share the same routing config. If a remote mounts PhoenixKit under a different prefix than the local site, override it:

config :phoenix_kit_sync, remote_url_prefix: "/custom"

Use "" (or "/") when the remote serves the sync API at the root (e.g. https://remote.example.com/sync/api/...).

Database

Table migrations are currently managed by PhoenixKit's core migration system. If the tables don't already exist, this package can create them automatically via PhoenixKitSync.Migration.up/0.

See docs/table_structure.md in the source repository for full schema documentation.

Future Plans

Auto-Sync Scheduling (Planned)

Connections have fields for auto-sync but the scheduler isn't implemented yet:

• auto_sync_enabled: Enable automatic periodic sync
• auto_sync_tables: Tables to sync automatically
• auto_sync_interval_minutes: How often to sync

Security Considerations

• Token Security: Tokens are SHA-256 hashed in the database and only shown once on creation; comparisons use Plug.Crypto.secure_compare/2 to defeat timing attacks
• Per-connection table authorization: Even with a valid token, every API endpoint that returns or accepts table data (list_tables, pull_data, table_schema, table_records) filters through the connection's excluded_tables blocklist and allowed_tables allowlist
• Parameterised SQL with identifier validation: Every dynamic table / column name interpolated into raw SQL passes through SchemaInspector.valid_identifier?/1 first; values always go through $N parameterised binds
• Optional Password: Additional password can be required per-transfer; refuses registration outright when incoming_mode = "require_password" but no password is configured (no silent bypass)
• IP Whitelisting: Restrict connections to specific IP addresses
• Time Restrictions: Allow connections only during specific hours
• Rate Limiting: Per-connection download / record limits enforced by Connections.validate_connection/2
• Audit Trail: Every state-changing operation lands a row in phoenix_kit_activities with actor_uuid, resource_type, resource_uuid, and a PII-safe metadata subset (no site_url, no auth_token_hash)

Troubleshooting

Connection Issues

1. Verify the token is correct and hasn't been regenerated
2. Check connection status is "active"
3. Verify IP is in whitelist (if configured)
4. Check time-of-day restrictions
5. Verify download/record limits haven't been exceeded

Transfer Failures

1. Check transfer history for error messages
2. Verify table is in allowed tables (if configured)
3. Check approval status if approval mode is enabled
4. Review server logs for detailed errors