Domo

Mix.install([:domo], force: true)

About

Elixir CIMethod TDDhex.pm version

A library to validate values of nested structs with their type spec t() and associated precondition functions.

Example apps

🔗 JSON parsing and validation example

🔗 Commanded + Domo combo used in Event Sourcing and CQRS app

🔗 Ecto + Domo combo in example_avialia app

🔗 TypedStruct + Domo combo in example_typed_integrations app

Description

Used in a struct's module, the library adds constructor, validation, and reflection functions. When called, constructor and validation functions guarantee the following:

If the conditions described above are not met, the constructor and validation functions return an error.

The business rule expressed via the precondition function can be shared across all structs referencing the appropriate type.

In terms of Domain Driven Design, types and associated precondition functions define the invariants relating structs to each other.

Tour

Run in Livebook

Let's say that we have a LineItem and PurchaseOrder structs with relating invariant that is the sum of line item amounts should be less then order's approved limit. That can be expressed like the following:

defmodule LineItem do
use Domo
defstruct amount: 0
@type t :: %__MODULE__{amount: non_neg_integer()}
end
defmodule PurchaseOrder do
use Domo
defstruct id: 1000, approved_limit: 200, items: []
@type id :: non_neg_integer()
precond(id: &(1000 <= &1 and &1 <= 5000))
@type t :: %__MODULE__{
id: id(),
approved_limit: pos_integer(),
items: [LineItem.t()]
}
precond(t: &validate_invariants/1)
defp validate_invariants(po) do
amounts = po.items |> Enum.map(& &1.amount) |> Enum.sum()
if amounts <= po.approved_limit do
:ok
else
{:error, "Sum of line item amounts (#{amounts}) should be <= to approved limit (#{po.approved_limit})."}
end
end
end

Then PurchaseOrder struct can be constructed consistently with functions generated by Domo like the following:

PurchaseOrder.new()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: []}}

The constructor function takes any Enumerable as the input value:

{:ok, po} = PurchaseOrder.new(%{approved_limit: 250})
{:ok, %PurchaseOrder{approved_limit: 250, id: 1000, items: []}}

It returns the descriptive keyword list if there is an error in input arguments. And it validates nested structs automatically:

PurchaseOrder.new(id: 500, items: [%LineItem{amount: -5}])
{:error,
[
items: "Invalid value [%LineItem{amount: -5}] for field :items of %PurchaseOrder{}.
Expected the value matching the [%LineItem{}] type.
Underlying errors:
- The element at index 0 has value %LineItem{amount: -5} that is invalid.
- Value of field :amount is invalid due to Invalid value -5 for field :amount
of %LineItem{}. Expected the value matching the non_neg_integer() type.",
id: "Invalid value 500 for field :id of %PurchaseOrder{}. Expected the
value matching the non_neg_integer() type. And a true value from
the precondition function \"&(1000 <= &1 and &1 <= 5000)\"
defined for PurchaseOrder.id() type."
]}

The returned errors are verbose and are intended for debugging purposes. See the Error messages for a user section below for more options.

The manually updated struct can be validated like the following:

po
|> Map.put(:items, [LineItem.new!(amount: 150)])
|> PurchaseOrder.ensure_type()
{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: [%LineItem{amount: 150}]}}

Domo returns the error if the precondition function attached to the t() type that validates invariants for the struct as a whole fails:

updated_po = %{po | items: [LineItem.new!(amount: 180), LineItem.new!(amount: 100)]}
PurchaseOrder.ensure_type(updated_po)
{:error, [t: "Sum of line item amounts should be <= to approved limit"]}

Getting the list of the required fields of the struct that have type other then nil or any is like that:

PurchaseOrder.required_fields()
[:approved_limit, :id, :items]

See the Callbacks section for more details about functions added to the struct.

Error messages for a user

It's possible to attach error messages to types with the precond macro to display them later to the user. To filter such kinds of messages, pass the maybe_filter_precond_errors: true option to Domo generated functions like that:

defmodule Book do
use Domo
defstruct [:title, :pages]
@type title :: String.t()
precond title: &(if String.length(&1) > 1, do: :ok, else: {:error, "Book title is required."})
@type pages :: pos_integer()
precond pages: &(if &1 > 2, do: :ok, else: {:error, "Book should have more then 3 pages. Given (#{&1})."})
@type t :: %__MODULE__{title: nil | title(), pages: nil | pages()}
end
defmodule Shelf do
use Domo
defstruct books: []
@type t :: %__MODULE__{books: [Book.t()]}
end
defmodule PublicLibrary do
use Domo
defstruct shelves: []
@type t :: %__MODULE__{shelves: [Shelf.t()]}
end
library = struct!(PublicLibrary, %{shelves: [struct!(Shelf, %{books: [struct!(Book, %{title: "", pages: 1})]})]})
PublicLibrary.ensure_type(library, maybe_filter_precond_errors: true)
{:error,
[
shelves: [
"Book title is required.",
"Book should have more then 3 pages. Given (1)."
]
]}

That output contains only a flattened list of precondition error messages from the deeply nested structure.

Custom constructor function

Sometimes a value for the struct's field can be generated during the construction. By default, Domo generates the new(!)/1 constructor functions for you, which can be composable with a custom one. You can create the custom constructor function with the same name by instructing Domo to use another one with gen_constructor_name option, like the following:

defmodule Foo do
use Domo, skip_defaults: true, gen_constructor_name: :_new
defstruct [:id, :token]
@type id :: non_neg_integer()
@type token :: String.t()
precond token: &byte_size(&1) == 8
@type t :: %__MODULE__{id: id(), token: token()}
def new(id) do
_new(id: id, token: random_string(8))
end
def new!(id) do
_new!(id: id, token: random_string(8))
end
defp random_string(length),
do: :crypto.strong_rand_bytes(length) |> Base.encode64() |> binary_part(0, length)
end
Foo.new!(15245)
%Foo{id: 15245, token: "e8K9wP0e"}

Compile-time and Run-time validations

At the project's compile-time, Domo performs the following checks:

At run-time, Domo validates structs matching their t() types.

Domo compiles TypeEnsurer module from struct's t() type to do all kinds of validations. There is a generated function with pattern matchings and guards for each struct's field. Constructor and validation functions of the struct delegate the work to the appropriate TypeEnsurer module.

After the compilation, the flow of control of the nested StructA validation can look like the following:

+--------------------+
| PurchaseOrder | +---------------------------+
| | | PurchaseOrder.TypeEnsurer |
| new(!)/1 ----------|--_ | |
| ensure_type(!)/1 --|-----|-> ensure_field_type/1 |
+--------------------+ | private functions |
+----|----------------------+
|
+-----------------------+ |
| LineItem | | +-------------------------+
| | | | LineItem.TypeEnsurer |
| new(!)/1 | | | |
| ensure_type(!)/1 | +--|-> ensure_field_type/1 |
+-----------------------+ | private functions |
+-------------------------+

In interactive mode (iex / livebook) Domo generates TypeEnsurer module dynamically as the last step of struct's module definition.

In mix compile mode Domo generates all TypeEnsurer modules after elixir compiler finishes its job. The generated code can be found in _build/MIX_ENV/domo_generated_code folder. However, that is for information purposes only. The following compilation will overwrite all changes there.

Depending types tracking

Let's suppose a structure field's type depends on a type defined in another module. When the latter type or its precondition changes, Domo recompiles the former module automatically to update its TypeEnsurer to keep the type validation up to date.

Domo tracks type-depending modules and touches appropriate files during compilation.

That works for any number of intermediate modules between the module defining the struct's field and the module defining the field's final type.

Integration with Ecto

Ecto schema changeset can be automatically validated to conform to t() type and associated preconditions. Then the changeset function can be shortened like the following:

defmodule Customer do
use Ecto.Schema
use Domo, skip_defaults: true
import Ecto.Changeset
import Domo.Changeset
schema "customers" do
field :first_name, :string
field :last_name, :string
field :birth_date, :date
timestamps()
end
@type t :: %__MODULE__{
first_name: String.t(),
last_name: String.t(),
birth_date: Date.t()
}
def changeset(changeset, attrs) do
changeset
|> cast(attrs, __schema__(:fields))
# Domo.Changeset defines validate_type/1 function.
|> validate_type()
end
end

The Domo validation comes with the price of about 1.5x times slower than the equivalent Ecto's validate_N functions.

See detailed example is in the example_avialia project.

Integration with libraries generating t() type for a struct

Domo is compatible with most libraries that generate t() type for a struct and an Ecto schema, f.e. typed_struct and typed_ecto_schema respectfully. Just use Domo in the module, and that's it.

An example is in the example_typed_integrations project.

TypedStruct's submodule generation with :module option currently is not supported.

Installation

To use Domo in a project, add the following line to mix.exs dependencies:

{:domo, "~> 1.5"}

And the following line to the project's mix.exs file:

compilers: [:domo_compiler] ++ Mix.compilers()

To exclude the generated TypeEnsurer modules from mix test --coverage add the following line, that works since Elixir v1.13, to the project's mix.exs:

test_coverage: [ignore_modules: [~r/\.TypeEnsurer$/]]

To avoid mix format putting extra parentheses around precond/1 macro call, add the following import to the .formatter.exs:

[
import_deps: [:domo]
]

Phoenix hot-reload

To enable Phoenix hot-reload for struct's type ensurers built by Domo, update the compilers in the mix.exs file like the following:

compilers: [:domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]

And add the following line to the endpoint's configuration in the config.exs file:

config :my_app, MyApp.Endpoint,
reloadable_compilers: [:phoenix, :domo_compiler] ++ Mix.compilers() ++ [:domo_phoenix_hot_reload]

Umbrella application

Add the Domo dependency and compilers config as mentioned in the section above to the mix.exs file for each app using Domo.

You may add the same domo compiler config line to the app itself and to the root umbrella's mix.exs to enable recompile command to work correctly for iex -S mix run in the root.

If you have the Phoenix hot-reload configured for one of the web apps in umbrella then the :domo_phoenix_hot_reload compiler should be added to all dependency apps used by the given web one.

Configuration

The options listed below can be set globally in the configuration with config :domo, option: value. The value given with use Domo, option: value overrides the global setting.

Run the Application.put_env(:domo, :verbose_in_iex, true) to enable verbose messages from domo in Interactive Elixir console.

Performance 🐢

Compilation-time

Library affects the project's full recompilation time almost insignificantly.

The compilation times for the business application with 38 structs (8 fields each on average) having 158 modules in total are the following:

Mode Average (by 3 measurements) Deviation
No Domo 14.826s 11.92
With Domo 15.711s 7.34
Comparison:
No Domo 14.826s
With Domo 15.711s - 1.06x slower

Run-time

The library ensures the correctness of data types at run-time, which comes with the computation price.

One of the standard tasks is translating the map with string values received as a form into a validated nested struct.

For benchmark, we use Album struct having many Track structs.

When comparing Domo generated new!/1 constructor function and Ecto changeset using a set of validate_.../2 functions internally, both have equivalent execution times. However, in the former case, the memory consumption is about 25% bigger.

In the case of Ecto changeset that uses Domo.Changeset.validate_type/1 function the computation takes about 1.5x times longer.

One of the benchmark results is shown below. The benchmark project is /benchmark_ecto_domo folder. You can run it with mix benchmark command.

Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.11.0
Erlang 24.3.3
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 8 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 36 s
Benchmarking Domo Album.new!/1...
Benchmarking Domo.Changeset validate_type/1...
Benchmarking Ecto.Changeset validate_.../1...
Name ips average deviation median 99th %
Domo Album.new!/1 5.31 188.25 ms ±1.79% 188.28 ms 196.57 ms
Ecto.Changeset validate_.../1 5.21 191.85 ms ±1.94% 191.00 ms 202.22 ms
Domo.Changeset validate_type/1 3.76 266.19 ms ±1.20% 266.58 ms 271.01 ms
Comparison:
Domo Album.new!/1 5.31
Ecto.Changeset validate_.../1 5.21 - 1.02x slower +3.59 ms
Domo.Changeset validate_type/1 3.76 - 1.41x slower +77.93 ms
Memory usage statistics:
Name Memory usage
Domo Album.new!/1 245.73 MB
Ecto.Changeset validate_.../1 186.59 MB - 0.76x memory usage -59.13956 MB
Domo.Changeset validate_type/1 238.69 MB - 0.97x memory usage -7.04444 MB
**All measurements for memory usage were the same**

Limitations

Parametrized types are not supported because it adds lots of complexity. Library returns {:type_not_found, :key} error for @type dict(key, value) :: [{key, value}] definition. Library returns error for type referencing parametrized type like @type field :: container(integer()).

Primitive types referencing themselves are not supported. Library returns an error for @type leaf :: leaf | nil definition. On the other hand, structs referencing themselves are supported. The library will build TypeEnsurer for the following definition @type t :: %__MODULE__{leaf: t | nil} and validate.

Migration

To migrate to a new version of Domo, please, clean and recompile the project with mix clean --deps && mix compile command.

Adoption

It's possible to adopt Domo library in the project having user-defined constructor functions named new/1 by refactoring to the Domo generated one. Here's how:

  1. Add :domo dependency to the project, configure compilers as described in the installation section
  2. Set the name of the Domo generated constructor function by adding config :domo, :gen_constructor_name, :_new option into the confix.exs file, to prevent conflict with user-defined constructor function name
  3. Add use Domo to existing struct, f.e. FirstStruct
  4. Change all struct building calls to be done with Domo generated function with the name set on step 3 f.e. FistStruct._new(%{...})
  5. Repeat for each struct in the project
  6. Remove original new/1 if it's not needed anymore and rename _new to new in the whole project
  7. Remove config :domo, :gen_constructor_name configuration because Domo generates constructor wiht new name by default.

Callbacks

Constructor, validation, and reflection functions added to the struct module using Domo.

new!/1/0

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

Raises an ArgumentError if conditions described above are not fulfilled.

This function will check if every given key-value belongs to the struct and raise KeyError otherwise.

new/2/1/0

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable in the shape of {:ok, struct_value}. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

If conditions described above are not fulfilled, the function returns an appropriate error in the shape of {:error, message_by_field}. message_by_field is a keyword list where the key is the name of the field and value is the string with the error message.

Keys in the enumerable that don't exist in the struct are automatically discarded.

Options

ensure_type!/1

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid. Raises an ArgumentError otherwise.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

ensure_type/2/1

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid in the shape of {:ok, struct}. Otherwise returns the error in the shape of {:error, message_by_field}.

Takes the same options as new/2.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

typed_fields/1/0

Returns the list of struct's fields defined with explicit types in its t() type spec.

Does not return meta fields with __underscored__ names and fields having any() type by default.

Includes fields that have nil type into the return list.

Options

required_fields/1/0

Returns the list of struct's fields having type others then nil or any().

Does not return meta fields with __underscored__ names.

Useful for validation of the required fields for emptiness. F.e. with validate_required/2 call in the Ecto changeset.

Options

Contributing

  1. Fork the repository and make a feature branch

  2. After implementing of the feature format the code with:

    mix format

    run linter and tests to ensure that all works as expected with:

    mix check || mix check --failed
  3. Make a PR to this repository

Changelog

v1.5.11

v1.5.10

v1.5.9 (2022-11-11)

v1.5.8 (2022-09-11)

v1.5.7 (2022-08-06)

v1.5.6 (2022-06-12)

v1.5.5 (2022-06-12)

v1.5.4 (2022-05-23)

v1.5.3 (2022-04-10)

v1.5.2 (2022-01-02)

v1.5.1 (2021-12-12)

v1.5.0 (2021-12-05)

Breaking change:

v1.4.1 (2021-11-16)

v1.4.0 (2021-11-15)

Breaking changes:

v1.3.4 (2021-10-13)

v1.3.3 (2021-10-07)

v1.3.2 (2021-09-18)

v1.3.1 (2021-08-19)

v1.3.0 (2021-08-15)

v1.2.9 (2021-08-09)

v1.2.8 (2021-07-15)

v1.2.7 (2021-07-05)

v1.2.6 (2021-06-21)

v1.2.5 (2021-06-14)

v1.2.4 (2021-06-07)

v1.2.3 (2021-05-31)

v1.2.2 (2021-05-05)

v1.2.1 (2021-04-25)

v1.2.0 (2021-04-12)

v0.0.x - v1.0.x (2020-06-20)

Roadmap

License

Copyright © 2020-2022 Ivan Rublev

This project is licensed under the MIT license.