RaftKV

An Elixir library to store key-value pairs in a distributed, fault-tolerant, self-adjusting data structure.

• API Documentation
• Hex package information

Feature & Design

• Each value can be arbitrary data structure.
  • Operation on a stored value must be given as an implementation of RaftKV.ValuePerKey behaviour.
• Built on top of raft_fleet.
  • Key-value pairs are sharded into multiple Raft consensus groups by hash partitioning.
• Based on number of keys, data size and current load, shards are automatically split/merged in a transparent manner.
• Designed for many key-value pairs and throughput.

Usage

Suppose we have the following callback module for simple key-value store.

defmodule KV do
  alias RaftKV.ValuePerKey
  @behaviour ValuePerKey

  @impl true
  def command(_previous_value, _size, _key, {:set, value}) do
    {:ok, 5, value, 0}
  end
  def command(_previous_value, _size, _key, :unset) do
    {:ok, 5, nil, 0}
  end

  @impl true
  def query(value, _size, _key, :get) do
    {value, 1}
  end

  #
  # API
  #
  def get(k) do
    case RaftKV.query(:kv, k, :get) do
      {:ok, v}                 -> v
      {:error, :key_not_found} -> nil
    end
  end

  def set(k, v) do
    {:ok, :ok} = RaftKV.command(:kv, k, {:set, v})
    :ok
  end

  def unset(k) do
    {:ok, :ok} = RaftKV.command(:kv, k, :unset)
    :ok
  end
end

Let's initialize :raft_kv and then register a keyspace.

RaftFleet.activate("some_zone")
RaftKV.init()
RaftKV.register_keyspace(:kv, KV, nil, %RaftKV.SplitMergePolicy{max_shards: 16, max_keys_per_shard: 100})

Now we can get/set values with arbitrary keys.

KV.get("foo")        # => nil
KV.set("foo", "bar") # => :ok
KV.get("foo")        # => "bar"

Initially there's only one shard.

RaftKV.reduce_keyspace_shard_names(:kv, [], &[&1 | &2]) # => [:kv_0]

When the number of key-value pairs exceeds the limit per shard (100) as follows...

Enum.each(1..200, fn i -> KV.set("#{i}", i) end)

then shards are automatically split. (Depending on the configurations it may take several minutes. See RaftKV.Config for more detail.)

RaftKV.reduce_keyspace_shard_names(:kv, [], &[&1 | &2]) # => [:kv_100663296, :kv_67108864, :kv_0]

Similarly, when you remove key-value pairs...

Enum.each(1..200, fn i -> KV.unset("#{i}") end)

the shards are automatically merged.

RaftKV.reduce_keyspace_shard_names(:kv, [], &[&1 | &2]) # => [:kv_0]

Shard splits and merges are transparent from client processes interacting with key-value pairs.