Welcome to Chord - a flexible and powerful Elixir library designed to simplify context management and delta tracking in your distributed or real-time applications.

Why Chord?

When you need a solution for real-time state synchronization, partial updates, and efficient cleanup, Chord strikes the perfect note! Here’s what makes Chord special:

• Seamless state sync: Keep your clients up-to-date with full context or delta-based updates.
• Customizable backend: Use ETS, Redis, or your own backend implementation.
• Flexible delta formatting: Define how your updates are structured.
• Periodic cleanup: Automatically clear stale contexts or deltas.
• Developer-friendly APIs: Simple, consistent, and easy-to-use APIs.
• Context export and restore: Export contexts to or restore them from external providers.
• Partial updates: Apply updates to specific fields within a context.
• Delta Tracking: Efficiently track and retrieve state changes.
• Telemetry integration: Built-in :telemetry events for monitoring and observability.
• Flexible architecture: Chord works in both stateful (via GenServer) and stateless modes (direct calls to backends like Redis or ETS). This flexibility makes it easier to adapt Chord to a variety of use cases.

Getting started

Install the library

Add Chord to your Mix dependencies:

def deps do
  [
    {:chord, "~> 0.3.0"}
  ]
end

If you plan to use the Redis backend, also add the optional dependencies:

{:redix, "~> 1.5"},
{:jason, "~> 1.4"}

Run:

mix deps.get

Configure Chord

Add your desired configuration in config/config.exs:

config :chord,
  backend: Chord.Backend.ETS,                     # Choose the backend (ETS, Redis, etc.)
  context_auto_delete: false,                     # Enables automatic deletion of old contexts
  context_ttl: 6 * 60 * 60,                       # Context time-to-live (follows `time_unit` format)
  delta_ttl: 24 * 60 * 60,                        # Delta time-to-live (follows `time_unit` format)
  delta_threshold: 100,                           # Maximum number of deltas to retain
  delta_formatter: Chord.Delta.Formatter.Default, # Default delta formatter; customizable
  time_provider: Chord.Utils.Time,                # Default time provider; customizable
  time_unit: :second,                             # Time unit (:second or :millisecond) for timestamps
  export_callback: nil,                           # Callback for persisting contexts
  context_external_provider: nil                  # Function for fetching external contexts

Explanation:

• context_auto_delete: Optional but recommended for efficient memory management.
  • If enabled, the following options must also be set:
    • context_ttl: Defines the time-to-live for contexts.
    • delta_ttl: Specifies the time-to-live for deltas.
    • delta_threshold: Determines the maximum number of deltas to retain.
• context_ttl & delta_ttl: Specify lifetimes for contexts and deltas. The values should align with the unit set in time_unit.
• delta_formatter: A default delta formatter is provided, but you can implement a custom formatter to suit your needs.
• time_provider: Responsible for generating timestamps. You can replace the default with a custom time provider.
• time_unit: Specifies the time unit for timestamps. Options are :second or :millisecond.
• export_callback: Define this callback to persist contexts from memory to an external storage solution.
• context_external_provider: Use this to retrieve contexts from external sources when needed.

How to use Chord

In Chord, a context is basically a container for state. The term “context” might mean different things in various fields, but in Chord, it specifically means a container for state. Here are some examples to explain this idea:

• In a chat application, a context could be a group chat, including its details (e.g., participants, topic), and messages.
• In a game session, a context might hold the game’s state, like player positions, scores, and progress.
• In a collaborative document editor, a context could be the document’s state, keeping track of edits, updates, and collaborators.

With this understanding of the term, let's look at some practical examples.

Setting a context

Define the global context and track changes with deltas.

Chord.set_context("user:369", %{status: "online", metadata: %{theme: "light", language: "en-US"}})
{:ok,
 %{
   context: %{
     version: 1,
     context: %{
       status: "online",
       metadata: %{language: "en-US", theme: "light"}
     },
     context_id: "user:369",
     inserted_at: 1737901562
   },
   delta: %{
     version: 1,
     context_id: "user:369",
     delta: %{
       status: %{value: "online", action: :added},
       metadata: %{
         language: %{value: "en-US", action: :added},
         theme: %{value: "light", action: :added}
       }
     },
     inserted_at: 1737901562
   }
 }}

Updating a context

Updates a portion of the global context associated with a specific identifier. This function allows for partial modifications without affecting the entire context.

Chord.update_context("user:369", %{metadata: %{theme: "dark"}})
{:ok,
 %{
   context: %{
     version: 2,
     context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
     context_id: "user:369",
     inserted_at: 1737901601
   },
   delta: %{
     version: 2,
     context_id: "user:369",
     delta: %{
       metadata: %{
         theme: %{value: "dark", action: :modified, old_value: "light"}
       }
     },
     inserted_at: 1737901601
   }
 }}

Getting a context

Fetches the current state for a specified identifier.

Chord.get_context("user:369")
{:ok,
 %{
   version: 2,
   context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
   context_id: "user:369",
   inserted_at: 1737901601
 }}

Synchronizing state

Synchronize the state for a given identifier. Depending on the version the client has, it will receive either the full context, only the changes (deltas), or a notification that there are no updates.

Chord.sync_context("user:369", nil)
{:full_context,
 %{
   version: 2,
   context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
   context_id: "user:369",
   inserted_at: 1737901601
 }}

Chord.sync_context("user:369", 1)
{:delta,
 %{
   version: 2,
   context_id: "user:369",
   delta: %{
     metadata: %{theme: %{value: "dark", action: :modified, old_value: "light"}}
   },
   inserted_at: 1737901601
 }}

Chord.sync_context("user:369", 2)
{:no_change, 2}

Exporting a context

Save the current context for a specific identifier to external storage using the configured export callback.

Defining the export callback

To enable the export functionality, you need to define a callback function in your application. This function will handle how the context is exported (e.g., saving it to a database). Here’s an example:

defmodule MyApp.ContextExporter do
  @moduledoc """
  Handles exporting contexts to external storage.
  """

  @spec export_context(map()) :: :ok | {:error, term()}
  def export_context(context_data) do
    %{context_id: context_id, version: verion, context: context} = context_data
    # Example: Save context_data to an external database or storage
    case ExternalStorage.save(context_id, context, version) do
      :ok -> :ok
      {:error, reason} -> {:error, reason}
    end
  end
end

Configure the export callback

Next, configure the export callback in your application’s environment. This tells Chord how to handle context exports.

# config/config.exs
config :chord, :export_callback, &MyApp.ContextExporter.export_context/1

Use Chord.export_context/1

Once the callback is configured, you can use function to export a specific context to external storage:

Chord.export_context("user:369")
:ok

Deleting a context

Removes the entire context and its associated deltas.

Chord.delete_context("user:369")
:ok

Restoring a context

Retrieve and restore a context from an external provider to the current backend.

Define the restore callback

First, define a module and function that will handle the logic for retrieving a context. For example:

defmodule MyApp.ContextRestorer do
  @moduledoc """
  Handles restoring contexts from external storage.
  """

  @spec restore_context(String.t()) :: {:ok, map()} | {:error, term()}
  def restore_context(context_id) do
    # Example: Retrieve the context from a database or other storage system
    case ExternalStorage.get(context_id) do
      {:ok, %{context: context, version: version}} -> {:ok, %{context: context, version: version}}
      {:error, reason} -> {:error, reason}
    end
  end
end

Configure the restore callback

Next, configure the restore callback in your application’s environment. This tells Chord how to handle context restoration:

# config/config.exs
config :chord, :context_external_provider, &MyApp.ContextRestorer.restore_context/1

Use Chord.restore_context/1

Once the callback is configured, you can use function to retrieve and restore a specific context:

Chord.restore_context("user:369")
{:ok,
 %{
   version: 10,
   context: %{source: "external storage provider"},
   inserted_at: 1737464001,
   context_id: "user:369"
 }}

Cleanup operations

Chord provides cleanup functionality to remove stale contexts and deltas. To enable and configure this feature, add the following settings to your application configuration:

Configuration options

config :chord,
  context_auto_delete: true, # Enable or disable auto-deletion of old contexts
  context_ttl: 6 * 60 * 60,  # Time-to-live for contexts
  delta_ttl: 24 * 60 * 60,   # Time-to-live for deltas
  delta_threshold: 100       # Number of delta versions to retain (optional)

How it works

• Context cleanup:

  • Set context_auto_delete: true to enable context cleanup.
  • Configure context_ttl to define how long contexts should remain in memory before being deleted.
  • When a context is deleted, all associated deltas are automatically cleaned up as well.

• Delta cleanup:

  • To clean deltas by age, set delta_ttl to specify the maximum time deltas should remain in memory.
  • To clean deltas by number, set delta_threshold to define the maximum number of deltas to retain.

Note: If the configured time unit is set to second, related configurations such as context_ttl and delta_ttl will also need to be specified in second to ensure consistency.

Example usage

Run the cleanup process manually with:

Chord.cleanup(limit: 50)

Managing the cleanup server

Start and manage the Cleanup Server for automated periodic cleanup:

{:ok, _pid} = Chord.start_cleanup_server(interval: :timer.minutes(30))
Chord.update_cleanup_interval(:timer.minutes(60))
Chord.update_cleanup_backend_opts(limit: 100)
Chord.stop_cleanup_server()

Customization

Backends

A backend refers to the underlying data storage mechanism responsible for managing and persisting context and delta data. Backends allow Chord to be flexible and adaptable to different storage solutions, whether in-memory, on disk, or external services.

Chord supports multiple backends out-of-the-box:

• ETS (In-Memory): No additional setup is required.
• Redis (Distributed): Requires a Redis instance, redix, and jason as dependencies.

Using Redis as a backend

To use Redis as the backend for Chord, follow these steps:

1. Add dependencies: Add redix and jason to your mix.exs (they are optional deps of Chord):

{:redix, "~> 1.5"},
{:jason, "~> 1.4"}

1. Start Redis: Ensure a Redis server is running.
2. Set up the Redis connection: Start a Redis connection process using the Redix library:

{:ok, _} = Redix.start_link("redis://localhost:6379", name: :my_redis)

1. Configure Chord to use Redis: Set the Redis client and backend in your application’s

# config/config.exs
config :chord,
  backend: Chord.Backend.Redis,
  redis_client: :my_redis

You can also implement your own backend by adhering to the Chord.Backend.Behaviour.

Delta formatters

Chord provides the ability to define custom delta formatters by implementing the Chord.Delta.Formatter.Behaviour. This feature is useful for tailoring how deltas (changes) are formatted to suit your application’s requirements.

Defining a custom delta formatter

To define a custom delta formatter, create a module that implements the Chord.Delta.Formatter.Behaviour:

defmodule MyApp.CustomFormatter do
  @moduledoc """
  A custom delta formatter for Chord, demonstrating how to implement the behavior.
  """

  @behaviour Chord.Delta.Formatter.Behaviour

  @impl true
  def format(delta, _context_id \\ nil) do
    flatten_delta(delta, [])
  end

  defp flatten_delta(delta, path) when is_map(delta) do
    Enum.flat_map(delta, fn {key, value} ->
      new_path = path ++ [key]

      if is_map(value) and Map.has_key?(value, :action) do
        [format_change(new_path, value)]
      else
        flatten_delta(value, new_path)
      end
    end)
  end

  defp format_change(path, %{action: action} = change) do
    base = %{key: path, action: action}

    case action do
      :added -> Map.put(base, :value, change.value)
      :modified -> Map.merge(base, %{old_value: change.old_value, value: change.value})
      :removed -> Map.put(base, :old_value, change.old_value)
    end
  end
end

Configuring Chord to use your delta formatter

Once you’ve defined your custom formatter, configure Chord to use it by setting it in the application environment:

# config/config.exs
config :chord, :delta_formatter, MyApp.CustomFormatter

Example usage

delta = %{
  a: %{
    f: %{value: "new", action: :added},
    b: %{
      c: %{
        d: %{value: "2", action: :modified, old_value: "1"},
        e: %{action: :removed, old_value: "3"}
      }
    }
  }
}

MyApp.CustomFormatter.format(delta)
[
  %{value: "new", key: [:a, :f], action: :added},
  %{value: "2", key: [:a, :b, :c, :d], action: :modified, old_value: "1"},
  %{key: [:a, :b, :c, :e], action: :removed, old_value: "3"}
]

Custom time provider

Chord allows you to define custom time provider by implementing the Chord.Utils.Time.Behaviour. This feature is useful for customizing time-based operations, such as timestamp generation and for mocking time in tests.

Defining a custom time provider

To define your custom time provider, create a module that implements the Chord.Utils.Time.Behaviour:

defmodule MyApp.CustomTimeProvider do
  @moduledoc """
  A custom time provider for Chord, demonstrating how to implement the behavior.
  """

  @behaviour Chord.Utils.Time.Behaviour

  @impl true
  def current_time(:second) do
    # Example: Use a custom logic for time in seconds
    DateTime.utc_now() |> DateTime.to_unix(:second)
  end

  @impl true
  def current_time(:millisecond) do
    # Example: Use a custom logic for time in milliseconds
    DateTime.utc_now() |> DateTime.to_unix(:millisecond)
  end
end

Configuring Chord to use your time provider

# config/config.exs
config :chord, :time_provider, MyApp.CustomTimeProvider

Telemetry

Chord emits :telemetry events for all core operations, making it easy to integrate with monitoring tools like Telemetry.Metrics, TelemetryMetricsPrometheus, or custom handlers.

Emitted events

Context events include %{context_id: context_id} in their metadata. All :stop events include %{duration: native_time} in measurements.

Example: attaching a handler

:telemetry.attach(
  "chord-context-set",
  [:chord, :context, :set, :stop],
  fn _event, measurements, metadata, _config ->
    IO.puts("set_context for #{metadata.context_id} took #{measurements.duration} native units")
  end,
  nil
)

Benchmark results: Redis and ETS performance

Chord has been tested to ensure solid performance in both Redis (single-node setup for now, with plans for distributed scenarios) and ETS (in-memory, single-node applications). Here’s how it performs under various scenarios:

Scenarios tested

1. Stateless operations

These scenarios simulate operations without maintaining a dedicated process per context. All updates, syncs and state modifications happen directly through the library’s API.

• Single context (50 participants): Represents a single group chat or meeting with 50 participants frequently updating their status, typing indicators, or syncing state.
• Multiple contexts (100 contexts): Simulates 100 independent group chats or meetings being updated simultaneously.

2. Stateful operations

These scenarios introduce a process per context (e.g., a GenServer for each group chat). Each participant interacts with this stateful process and the process uses Chord’s API to manage context.

• Single context (50 participants): A single group chat or meeting managed by a GenServer, handling frequent updates and syncs from 50 participants.
• Multiple contexts (100 contexts): Simulates 100 group chats or meetings, each managed by its own GenServer, handling participant interactions.

Results

ETS Backend (In-Memory, Single-Node)

Redis Backend (Single-Node)

Note: Redis benchmarks were conducted in a single-node configuration to evaluate baseline performance. A distributed Redis setup would allow benchmarking its true scalability for high-throughput, multi-context scenarios.

Key insights

ETS:

• Sub-millisecond single-context performance: Stateless operations with 50 participants achieve 1,610 ops/sec at 0.62 ms average, thanks to O(1) lookups and atomic upserts.
• Stateful nearly matches stateless: GenServer overhead is minimal for single contexts (1,260 vs 1,610 ops/sec), making stateful architecture viable without significant performance cost.
• Multi-context scaling: 100 contexts at 77 ops/sec (stateful) and 51 ops/sec (stateless) with tail latencies under 26 ms.

Redis:

• Strong single-context throughput: 480 ops/sec for stateless operations with cursor-based SCAN and optimized serialization.
• Consistent multi-context performance: Both stateful (32.5 ops/sec) and stateless (30.6 ops/sec) multi-context scenarios show roughly equal throughput.
• Tail latency: 99th percentile latency stays under 45 ms even for 100-context workloads.

Choosing between stateless and stateful

Stateless:

• Directly interacts with Chord’s API, bypassing the need for per-context processes.
• Best for: High-concurrency scenarios where multiple clients update a single shared context.

Stateful:

• Manages a dedicated GenServer per context (e.g., per group chat or meeting).
• Best for: Scenarios requiring additional application-level state or business logic. The performance gap between stateful and stateless is minimal for single-context workloads.

Device Information

Benchmark Suite Configuration:

• Warmup: 2 seconds
• Execution Time: 5 seconds
• Parallel: 1

Contributing

Contributions from the community are welcome to make Chord even better! Whether it's fixing bugs, improving documentation, or adding new features, your help is greatly appreciated.

How to contribute

1. Fork the repository.
2. Create a new branch for your changes.
3. Make your changes and test them thoroughly.
4. Submit a pull request with a clear description of your changes.

Feel free to open issues for discussion or if you need help. Together, we can build something amazing!

Testing & Quality

Chord comes with a comprehensive quality suite. Run the full checks with:

mix test                      # Run the test suite
mix format --check-formatted  # Check code formatting
mix compile --warnings-as-errors  # Compile with strict warnings
mix dialyzer                  # Run static type analysis

All checks are also enforced in CI via GitHub Actions.

🎵 "Let Chord orchestrate your state management with precision and elegance."