Snowflakex

Snowflakex (pronounced "snowflakes") is a service for generating unique ID numbers at high scale with some simple guarantees.

It is directly modeled after Twitter's Snowflake.

Installation

Add snowflakex to your list of dependencies in mix.exs:

def deps do
  [
    {:snowflakex, "~> 1.2"}
  ]
end

Then configure a machine id between 0 and 1023 that is unique for every node generating IDs, for example in config/runtime.exs:

config :snowflakex, machine_id: 123

The application refuses to start without a valid machine id.

Usage

iex> Snowflakex.new()
{:ok, 1234567890123456789}

Snowflakex.new/0 returns either {:ok, snowflake} or {:error, message}, where the latter only occurs when the system clock moved backwards.

iex> Snowflakex.new!()
1234567890123456789

Snowflakex.new!/0 returns the snowflake directly or raises a Snowflakex.ClockError holding the error message and the remaining time in milliseconds.

iex> Snowflakex.decompose(4_194_476_039)
%{timestamp: 1_480_927_902_749, machine_id: 42, sequence: 7}

Snowflakex.decompose/1 splits a snowflake into its components. The timestamp is given in milliseconds since the Unix epoch.

Requirements

Performance

Minimum 10k IDs per second per node
Response rate <= 2ms

Uncoordinated

For high availability within and across server application instances, nodes generating IDs should not have to coordinate with each other.

(Roughly) Time Ordered

We can guarantee that the ID numbers will be k-sorted (see here and here) within a reasonable bound (we're promising 1s, but shooting for 10's of ms).

Directly Sortable

The IDs should be sortable without loading the full objects that they represent. This sorting should be the above ordering.

Compact

There are many otherwise reasonable solutions to this problem that require 128-bit numbers. To accommodate most systems, we keep our IDs under 64 bits.

Highly Available

The ID generation scheme should be at least as available as the related services.

Solution

The ID numbers are 64-bit integers composed of:

Bits	Field	Description
1	Sign	Unused, always `0`
41	Timestamp	Milliseconds since the custom epoch `2016-12-05T08:51:41.749Z`, lasting 69 years
10	Machine id	Configured per node, allowing up to 1024 machines
12	Sequence	Rolls over every 4096 IDs per machine, with protection against rolling over within the same millisecond

System Clock Dependency

You should use NTP to keep your system clock accurate. Snowflakex protects from non-monotonic clocks, i.e. clocks that run backwards. If your clock is running fast and NTP tells it to repeat a few milliseconds, Snowflakex will refuse to generate IDs until a time that is after the last time we generated an ID. Even better, run in a mode where NTP won't move the clock backwards. See Dovecot's notes on time synchronization for tips on how to do this.

License

Licensed under the Apache License, Version 2.0.