Welcome to Chord - a flexible and powerful Elixir library designed to simplify context management and delta tracking in your distributed or real-time applications.

Why Chord?

When you need a solution for real-time state synchronization, partial updates, and efficient cleanup, Chord strikes the perfect note! Here’s what makes Chord special:

• Seamless state sync: Keep your clients up-to-date with full context or delta-based updates.
• Customizable backend: Use ETS, Redis, or your own backend implementation.
• Flexible delta formatting: Define how your updates are structured.
• Periodic cleanup: Automatically clear stale contexts or deltas.
• Developer-friendly APIs: Simple, consistent, and easy-to-use APIs.
• Context export and restore: Export contexts to or restore them from external providers.
• Partial updates: Apply updates to specific fields within a context.
• Delta Tracking: Efficiently track and retrieve state changes.
• Flexible architecture: Chord works in both stateful (via GenServer) and stateless modes (direct calls to backends like Redis or ETS). This flexibility makes it easier to adapt Chord to a variety of use cases.

🚀 Getting started

Install the library

Add Chord to your Mix dependencies:

def deps do
  [
    {:chord, "~> 0.1.4"}
  ]
end

Run:

mix deps.get

Configure Chord

Add your desired configuration in config/config.exs:

config :chord,
  backend: Chord.Backend.ETS,                     # Choose your backend (Redis, ETS, etc.)
  context_auto_delete: false,                     # Enable or disable auto-deletion of old contexts
  context_ttl: 6 * 60 * 60,                       # Time-to-live for contexts
  delta_ttl: 24 * 60 * 60,                        # Time-to-live for deltas
  delta_threshold: 100,                           # Number of deltas to retain
  delta_formatter: Chord.Delta.Formatter.Default, # Format for deltas
  time_provider: Chord.Utils.Time,                # Time provider for consistent timestamps
  export_callback: nil,                           # Optional: Define a callback for exporting contexts
  context_external_provider: nil                  # Optional: Define a function for fetching external contexts

How to use Chord

In Chord, a context is basically a container for state. The term “context” might mean different things in various fields, but in Chord, it specifically means a container for state. Here are some examples to explain this idea:

• In a chat application, a context could be a group chat, including its details (e.g., participants, topic), and messages.
• In a game session, a context might hold the game’s state, like player positions, scores, and progress.
• In a collaborative document editor, a context could be the document’s state, keeping track of edits, updates, and collaborators.

With this understanding of the term, let's look at some practical examples.

Setting a context

Define the global context and track changes with deltas.

Chord.set_context("user:123", %{status: "online", metadata: %{theme: "light", language: "en-US"}})
{:ok,
 %{
   context: %{
     version: 1,
     context: %{
       status: "online",
       metadata: %{language: "en-US", theme: "light"}
     },
     inserted_at: 1737570501,
     context_id: "user:123"
   },
   delta: %{
     version: 1,
     delta: %{
       status: %{value: "online", action: :added},
       metadata: %{value: %{language: "en-US", theme: "light"}, action: :added}
     },
     inserted_at: 1737570501,
     context_id: "user:123"
   }
 }}

Updating a context

Updates a portion of the global context associated with a specific identifier. This function allows for partial modifications without affecting the entire context.

Chord.update_context("user:123", %{metadata: %{theme: "dark"}})
{:ok,
 %{
   context: %{
     version: 2,
     context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
     inserted_at: 1737570583,
     context_id: "user:123"
   },
   delta: %{
     version: 2,
     delta: %{
       metadata: %{
         theme: %{value: "dark", action: :modified, old_value: "light"}
       }
     },
     inserted_at: 1737570583,
     context_id: "user:123"
   }
 }}

Getting a context

Fetches the current state for a specified identifier.

Chord.get_context("user:123")
{:ok,
 %{
   version: 2,
   context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
   inserted_at: 1737570583,
   context_id: "user:123"
 }}

Synchronizing state

Synchronize the state for a given identifier. Depending on the version the client has, it will receive either the full context, only the changes (deltas), or a notification that there are no updates.

Chord.sync_context("user:123", nil)
{:full_context,
 %{
   version: 2,
   context: %{status: "online", metadata: %{language: "en-US", theme: "dark"}},
   inserted_at: 1737570583,
   context_id: "user:123"
 }}

Chord.sync_context("user:123", 1)
{:delta,
 %{
   version: 2,
   delta: %{
     metadata: %{theme: %{value: "dark", action: :modified, old_value: "light"}}
   },
   inserted_at: 1737570583,
   context_id: "user:123"
 }}

Chord.sync_context("user:123", 2)
{:no_change, 2}

Exporting a context

Save the current context for a specific identifier to external storage using the configured export callback.

Defining the export callback

To enable the export functionality, you need to define a callback function in your application. This function will handle how the context is exported (e.g., saving it to a database). Here’s an example:

defmodule MyApp.ContextExporter do
  @moduledoc """
  Handles exporting contexts to external storage.
  """

  @spec export_context(map()) :: :ok | {:error, term()}
  def export_context(context_data) do
    %{context_id: context_id, version: verion, context: context} = context_data
    # Example: Save context_data to an external database or storage
    case ExternalStorage.save(context_id, context, version) do
      :ok -> :ok
      {:error, reason} -> {:error, reason}
    end
  end
end

Configure the export callback

Next, configure the export callback in your application’s environment. This tells Chord how to handle context exports.

# config/config.exs
config :chord, :export_callback, &MyApp.ContextExporter.export_context/1

Use Chord.export_context/1

Once the callback is configured, you can use function to export a specific context to external storage:

Chord.export_context("user:123")
:ok

Deleting a context

Removes the entire context and its associated deltas.

Chord.delete_context("user:123")
:ok

Restoring a context

Retrieve and restore a context from an external provider to the current backend.

Define the restore callback

First, define a module and function that will handle the logic for retrieving a context. For example:

defmodule MyApp.ContextRestorer do
  @moduledoc """
  Handles restoring contexts from external storage.
  """

  @spec restore_context(String.t()) :: {:ok, map()} | {:error, term()}
  def restore_context(context_id) do
    # Example: Retrieve the context from a database or other storage system
    case ExternalStorage.get(context_id) do
      {:ok, %{context: context, version: version}} -> {:ok, %{context: context, version: version}}
      {:error, reason} -> {:error, reason}
    end
  end
end

Configure the restore callback

Next, configure the restore callback in your application’s environment. This tells Chord how to handle context restoration:

# config/config.exs
config :chord, :context_external_provider, &MyApp.ContextRestorer.restore_context/1

Use Chord.restore_context/1

Once the callback is configured, you can use function to retrieve and restore a specific context:

Chord.restore_context("user:1234")
{:ok,
 %{
   version: 10,
   context: %{source: "external storage provider"},
   inserted_at: 1737464001,
   context_id: "user:1234"
 }}

Cleanup operations

Chord provides cleanup functionality to remove stale contexts and deltas. To enable and configure this feature, add the following settings to your application configuration:

Configuration options

config :chord,
  context_auto_delete: true, # Enable or disable auto-deletion of old contexts
  context_ttl: 6 * 60 * 60,  # Time-to-live for contexts (in seconds)
  delta_ttl: 24 * 60 * 60,   # Time-to-live for deltas (optional, in seconds)
  delta_threshold: 100       # Number of delta versions to retain (optional)

How it works

• Context cleanup:

  • Set context_auto_delete: true to enable context cleanup.
  • Configure context_ttl to define how long contexts should remain in memory before being deleted. The value must be in seconds, as timestamps are created with second-level precision.
  • When a context is deleted, all associated deltas are automatically cleaned up as well.

• Delta cleanup:

  • To clean deltas by age, set delta_ttl to specify the maximum time deltas should remain in memory.
  • To clean deltas by number, set delta_threshold to define the maximum number of deltas to retain.

Example usage

Run the cleanup process manually with:

Chord.cleanup(limit: 50)

Future plans

In the future, you’ll be able to configure the time unit (e.g., seconds, milliseconds), which will be used when generating timestamps. This configuration will apply to the function responsible for providing the current time, primarily used when setting inserted_at during data storage in memory.

If the configured time unit is set to seconds, related configurations such as context_ttl and delta_ttl will also need to be specified in seconds to ensure consistency.

Managing the cleanup server

Start and manage the Cleanup Server for automated periodic cleanup:

{:ok, _pid} = Chord.start_cleanup_server(interval: :timer.minutes(30))
Chord.update_cleanup_interval(:timer.minutes(60))
Chord.update_cleanup_backend_opts(limit: 100)
Chord.stop_cleanup_server()

Customization

Backends

Chord supports multiple backends out-of-the-box:

• ETS (In-Memory): No additional setup is required.
• Redis (Distributed): Requires a Redis instance and some configuration.

Using Redis as a backend

To use Redis as the backend for Chord, follow these steps:

1. Start Redis: Ensure a Redis server is running.
2. Set up the Redis connection: Start a Redis connection process using the Redix library, which is included with Chord:

{:ok, _} = Redix.start_link("redis://localhost:6379", name: :my_redis)

1. Configure Chord to use Redis: Set the Redis client and backend in your application’s

# config/config.exs
config :chord,
  backend: Chord.Backend.Redis,
  redis_client: :my_redis

You can also implement your own backend by adhering to the Chord.Backend.Behaviour.

Delta formatters

Chord provides the ability to define custom delta formatters by implementing the Chord.Delta.Formatter.Behaviour. This feature is useful for tailoring how deltas (changes) are formatted to suit your application’s requirements.

Defining a custom delta formatter

To define a custom delta formatter, create a module that implements the Chord.Delta.Formatter.Behaviour:

defmodule MyApp.CustomFormatter do
  @moduledoc """
  A custom delta formatter for Chord, demonstrating how to implement the behavior.
  """

  @behaviour Chord.Delta.Formatter.Behaviour

  @impl true
  def format(delta, context_id) do
    timestamp = DateTime.utc_now() |> DateTime.to_iso8601()

    Enum.map(delta, fn {key, change} ->
      base = %{
        timestamp: timestamp,
        context_id: context_id,
        field: key,
        action: change.action
      }

      case change.action do
        :added ->
          Map.put(base, :new_value, change.value)

        :modified ->
          Map.merge(base, %{old_value: change.old_value, new_value: change.value})

        :removed ->
          base
      end
    end)
  end
end

Configuring Chord to use your delta formatter

Once you’ve defined your custom formatter, configure Chord to use it by setting it in the application environment:

# config/config.exs
config :chord, :delta_formatter, MyApp.CustomFormatter

Example usage

delta = %{
  a: %{action: :added, value: 1},
  b: %{action: :modified, old_value: 2, value: 3},
  c: %{action: :removed}
}

context_id = "game:42"

formatted_delta = MyApp.CustomFormatter.format(delta, context_id)

IO.inspect(formatted_delta)
[
  %{
    timestamp: "2025-01-22T15:00:00Z",
    context_id: "game:42",
    field: :a,
    action: :added,
    new_value: 1
  },
  %{
    timestamp: "2025-01-22T15:00:00Z",
    context_id: "game:42",
    field: :b,
    action: :modified,
    old_value: 2,
    new_value: 3
  },
  %{
    timestamp: "2025-01-22T15:00:00Z",
    context_id: "game:42",
    field: :c,
    action: :removed
  }
]

Custom time provider

Chord allows you to define custom time provider by implementing the Chord.Utils.Time.Behaviour. This feature is useful for customizing time-based operations, such as timestamp generation and for mocking time in tests.

Defining a custom time provider

To define your custom time provider, create a module that implements the Chord.Utils.Time.Behaviour:

defmodule MyApp.CustomTimeProvider do
  @moduledoc """
  A custom time provider for Chord, demonstrating how to implement the behavior.
  """

  @behaviour Chord.Utils.Time.Behaviour

  @impl true
  def current_time(:second) do
    # Example: Use a custom logic for time in seconds
    DateTime.utc_now() |> DateTime.to_unix(:second)
  end

  @impl true
  def current_time(:millisecond) do
    # Example: Use a custom logic for time in milliseconds
    DateTime.utc_now() |> DateTime.to_unix(:millisecond)
  end
end

Configuring Chord to use your time provider

# config/config.exs
config :chord, :time_provider, MyApp.CustomTimeProvider

Benchmark results: Redis and ETS performance

Chord has been tested to ensure solid performance in both Redis (single-node setup for now, with plans for distributed scenarios) and ETS (in-memory, single-node applications). Here’s how it performs under various scenarios:

Scenarios tested

1. Stateless operations

These scenarios simulate operations without maintaining a dedicated process per context. All updates, syncs and state modifications happen directly through the library’s API.

• Single context (50 participants): Represents a single group chat or meeting with 50 participants frequently updating their status, typing indicators, or syncing state.
• Multiple contexts (100 contexts): Simulates 100 independent group chats or meetings being updated simultaneously.

2. Stateful operations

These scenarios introduce a process per context (e.g., a GenServer for each group chat). Each participant interacts with this stateful process and the process uses Chord’s API to manage context.

• Single context (50 participants): A single group chat or meeting managed by a GenServer, handling frequent updates and syncs from 50 participants.
• Multiple contexts (100 contexts): Simulates 100 group chats or meetings, each managed by its own GenServer, handling participant interactions.

Results

Redis backend (single node)

Note: Redis benchmarks were conducted in a single-node configuration to evaluate baseline performance. While Redis is designed for distributed systems, a fully distributed environment is not yet implemented in the benchmark script. Plans are underway to expand the benchmarking script to support distributed scenarios.

ETS Backend (In-Memory, Single-Node)

Key insights

Redis:

• Single-node performance: Reflects the baseline for Redis's capability, with potential for distributed scaling in the future.
• Stateless operations Outperform stateful ones when multiple clients update a single context concurrently. The absence of GenServer synchronization overhead makes Redis particularly well-suited for high-frequency, multi-client updates to shared contexts.
• Stateful performance bottlenecks: Syncing multiple contexts (100) causes significant performance degradation due to process synchronization and network overhead.
• Future improvements: A distributed Redis setup would allow benchmarking its true scalability and potential for handling high-throughput, multi-context scenarios.

ETS:

• Optimal for single-node applications: Outshines Redis in single-node scenarios, particularly in stateless operations where 50 participants concurrently update a single context, achieving 230.61 ops/sec with just 4.34 ms latency.
• GenServer overhead: Stateful operations see reduced performance due to process-based synchronization, especially with many contexts (e.g., 100).
• Scalability limitations:: While ETS is efficient for localized, single-node setups, it struggles with multi-context workloads, where its performance drops to 4.69 ops/sec for stateless and 5.66 ops/sec for stateful scenarios.

Choosing between stateless and stateful

Stateless:

• Directly interacts with Chord’s API, bypassing the need for per-context processes.
• Best for: High-concurrency scenarios where multiple clients update a single shared context. Performance degrades under high multi-context workloads, particularly in Redis.

Stateful:

• Manages a dedicated GenServer per context (e.g., per group chat or meeting).
• Best for: Scenarios requiring additional application-level state or business logic. However, high-concurrency and multi-context workloads may lead to significant performance degradation.

Device Information

Benchmark Suite Configuration:

• Warmup: 2 seconds
• Execution Time: 5 seconds
• Parallel: 1
• Inputs: Data

Contributing

Contributions from the community are welcome to make Chord even better! Whether it's fixing bugs, improving documentation, or adding new features, your help is greatly appreciated.

How to contribute

1. Fork the repository.
2. Create a new branch for your changes.
3. Make your changes and test them thoroughly.
4. Submit a pull request with a clear description of your changes.

Feel free to open issues for discussion or if you need help. Together, we can build something amazing!

Testing

Chord comes with a robust suite of tests to ensure reliability. Run tests with:

mix test

🎵 "Let Chord orchestrate your state management with precision and elegance."