StatifierDatamodel

The datamodel document and what can be decided from it, with no dependency on the block editor, the compiler, or the UI.

A datamodel document is a host's typed description of the data universe an author writes conditions against. It has three scopes - global, local, event - each carrying entries with a name, an absolute dotted path, a type and a label; and a types key of named record and shape declarations, each with ordered fields carrying name, type and required?. This package is the reader of that document, and the home of every question that can be answered from the document alone.

The contract is ADR-0001; the document shape and the index are re-homed here from statifier_blocks' ADR-0006, and ADR-0001 is the record of the re-homing.

Installation

def deps do
  [
    {:statifier_datamodel, "~> 0.4"}
  ]
end

What it answers

Every example below is a doctest: test/readme_test.exs runs this file, so a snippet that stops being true fails the suite. The worked domain is credit-card processing, as it is everywhere in this family.

The index

StatifierDatamodel.Index.index/1 admits a decoded document and returns the path/type index over it. Every entry contributes its own path at every nesting depth, in document order, and nothing else does. Admission is a total normalizer: a map carrying a list under "scopes" is a document, and anything else is nil - so not a document stays distinguishable from a document declaring nothing.

iex> alias StatifierDatamodel.Index
iex> document = %{
...>   "version" => 1,
...>   "scopes" => [
...>     %{"scope" => "local", "label" => "Chart-local", "entries" => [
...>       %{"name" => "amount_cents", "path" => "amount_cents",
...>         "type" => "integer", "label" => "Amount (minor units)"},
...>       %{"name" => "card", "path" => "card", "type" => "object",
...>         "label" => "Card", "fields" => [
...>           %{"name" => "brand", "path" => "card.brand", "type" => "string",
...>             "label" => "Brand",
...>             "one_of" => ["visa", "mastercard", "amex"]},
...>           %{"name" => "expires_on", "path" => "card.expires_on",
...>             "type" => "date", "label" => "Expires on"}]}]}]}
iex> index = Index.index(document)
iex> index.order
["amount_cents", "card", "card.brand", "card.expires_on"]
iex> Index.type(index, "card.expires_on")
:date
iex> Index.declared?(index, "card.cvv")
false
iex> Index.index(["amount_cents"])
nil

The declared paths

StatifierDatamodel.Document is the shorthand for a consumer that holds a document rather than an index: it admits its argument through the index and answers totally either way. declared_paths/1 is the projection an editor's undeclared-path advisory reads; an empty document projects to an empty set, and only a non-document is nil.

iex> alias StatifierDatamodel.Document
iex> document = %{"scopes" => [%{"scope" => "local", "entries" => [
...>   %{"name" => "card", "path" => "card", "type" => "object", "fields" => [
...>     %{"name" => "last4", "path" => "card.last4", "type" => "string"},
...>     %{"name" => "brand", "path" => "card.brand", "type" => "string",
...>       "one_of" => ["visa", "mastercard", "amex"]}]}]}]}
iex> Document.declared_paths(document)
MapSet.new(["card", "card.brand", "card.last4"])
iex> Document.candidates_under(document, "card")
["card.last4", "card.brand"]
iex> Document.declared_values(document)
%{"card.brand" => ["visa", "mastercard", "amex"]}
iex> Document.declared_paths(%{"scopes" => []})
MapSet.new([])
iex> Document.declared_paths(["card"])
nil

The declared types and the read check

A record is a fact about what a write puts at a path; a shape is a constraint a read places on one. StatifierDatamodel.Declarations indexes the document's types key to name -> declaration, and StatifierDatamodel.Types decides the read check over it: unknown is permissive both ways, then identity, then a record read as a shape is admitted when the record's fields cover the shape's required set. Covering a required shape field takes a record field of the same name that the record also declares required: an optional field has not promised the value, so it does not cover. Identity is nominal - there is no structural widening between two records. satisfies/3 returns the reason a consumer renders; satisfies?/3 is the same check as a boolean.

iex> alias StatifierDatamodel.{Declarations, Types}
iex> declarations = Declarations.from_document(%{"types" => [
...>   %{"name" => "cards.credit_txn", "kind" => "record",
...>     "label" => "Credit transaction", "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => true},
...>       %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]},
...>   %{"name" => "Settleable", "kind" => "shape", "label" => "Settleable",
...>     "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => true},
...>       %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]},
...>   %{"name" => "Refundable", "kind" => "shape", "label" => "Refundable",
...>     "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "settled_at", "type" => "datetime", "required?" => true}]}]})
iex> Types.satisfies(declarations, {:declared, "cards.credit_txn"}, {:declared, "Settleable"})
:covers
iex> Types.satisfies(declarations, {:declared, "cards.credit_txn"}, {:declared, "Refundable"})
{:missing, ["settled_at"]}
iex> Types.satisfies?(declarations, {:declared, "cards.credit_txn"}, {:declared, "Refundable"})
false
iex> Types.satisfies(declarations, :date, :date)
:identical

Drop the required? on the record's authorized_at and the same read stops being satisfied, naming that field the way an absent one is named. The fix in the document is one key: mark the field required where the record does promise the value.

iex> alias StatifierDatamodel.{Declarations, Types}
iex> loose = Declarations.from_document(%{"types" => [
...>   %{"name" => "cards.credit_txn", "kind" => "record",
...>     "label" => "Credit transaction", "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => true},
...>       %{"name" => "authorized_at", "type" => "datetime"}]},
...>   %{"name" => "Settleable", "kind" => "shape", "label" => "Settleable",
...>     "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => true},
...>       %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]}]})
iex> Types.satisfies(loose, {:declared, "cards.credit_txn"}, {:declared, "Settleable"})
{:missing, ["authorized_at"]}

An inline, unnamed shape

A consumer holding a value the host never declared - a fan-out's collected envelope, a block's computed summary - says what it holds with {:shape, members}, where each member carries name, type and required?. It is read the same way a declaration is: a record covers it member-wise, it covers a declared shape, and two inline shapes compare structurally. It never satisfies a declared record, whose identity is nominal, and a declared shape held covers nothing but itself. Identity for the arm is member-set-wise, so member order is a rendering decision and not a difference. There is no document spelling for one: parse/2 reads only strings, and an inline shape enters as an argument to the read check.

iex> alias StatifierDatamodel.{Declarations, Types}
iex> declarations = Declarations.from_document(%{"types" => [
...>   %{"name" => "ChunkSummary", "kind" => "shape", "label" => "Chunk summary",
...>     "fields" => [
...>       %{"name" => "authorized_count", "type" => "integer", "required?" => true},
...>       %{"name" => "declined_count", "type" => "integer", "required?" => true}]}]})
iex> summary = {:shape, [
...>   %{name: "authorized_count", type: :integer, required?: true},
...>   %{name: "declined_count", type: :integer, required?: true}]}
iex> envelope = {:shape, [
...>   %{name: "index", type: :integer, required?: true},
...>   %{name: "status", type: :string, required?: true},
...>   %{name: "donedata", type: summary, required?: false}]}
iex> Types.satisfies(declarations, summary, {:declared, "ChunkSummary"})
:covers
iex> Types.satisfies(declarations, envelope, {:declared, "ChunkSummary"})
{:missing, ["authorized_count", "declined_count"]}
iex> Types.to_string(summary)
"{authorized_count: integer, declined_count: integer}"

The envelope's own members are index, status and donedata: the summary is one member down, and this package widens nothing to find it.

An entry typed by a declaration

An entry's type, and a list entry's item_type, may name a declaration instead of one of the nine types. The reference is nominal - the index carries {:declared, name}, not a copy of the fields - and the entry then contributes the declaration's fields beneath its own path, exactly as an inlined object entry contributes its fields. A name the types key does not declare stays unknown, as it always has.

iex> alias StatifierDatamodel.Index
iex> index = Index.index(%{"scopes" => [
...>   %{"scope" => "local", "entries" => [
...>     %{"name" => "txn", "path" => "txn", "type" => "cards.credit_txn",
...>       "label" => "Transaction"},
...>     %{"name" => "note", "path" => "note", "type" => "cards.nothing"}]}],
...>   "types" => [
...>     %{"name" => "cards.credit_txn", "kind" => "record",
...>       "label" => "Credit transaction", "fields" => [
...>         %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>         %{"name" => "card", "type" => "cards.card", "required?" => true}]},
...>     %{"name" => "cards.card", "kind" => "record", "label" => "Card",
...>       "fields" => [
...>         %{"name" => "brand", "type" => "string",
...>           "one_of" => ["visa", "mastercard", "amex"]}]}]})
iex> index.order
["txn", "txn.amount_cents", "txn.card", "txn.card.brand", "note"]
iex> Index.type(index, "txn.card")
{:declared, "cards.card"}
iex> Index.type(index, "note")
nil
iex> Index.path_types(index)
%{
  "txn.amount_cents" => :number,
  "txn.card.brand" => {:one_of, ["visa", "mastercard", "amex"]}
}

txn and txn.card are absent from the value kinds for the same reason an object entry is: neither is a value the expression language has a kind for. A cycle between declarations discharges rather than recurring, so index/1 stays total over any document a host can write.

Compatibility of a redefined declaration

StatifierDatamodel.Compatibility.breaks/2 is given the declaration a name had and the declaration replacing it, and lists every way the new one narrows the old one - every reason a read that held under the old might not hold under the new - ordered by field name. The list is empty when the redefinition takes nothing away; a name declared on neither side is :error, never an empty list.

iex> alias StatifierDatamodel.{Compatibility, Declarations}
iex> was = Declarations.from_document(%{"types" => [
...>   %{"name" => "cards.credit_txn", "kind" => "record",
...>     "label" => "Credit transaction", "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => true},
...>       %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]}]})
iex> now = Declarations.from_document(%{"types" => [
...>   %{"name" => "cards.credit_txn", "kind" => "record",
...>     "label" => "Credit transaction", "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => false}]}]})
iex> {:ok, old} = Declarations.fetch(was, "cards.credit_txn")
iex> {:ok, new} = Declarations.fetch(now, "cards.credit_txn")
iex> Compatibility.breaks(old, new)
[{:field_removed, "authorized_at"}, {:made_optional, "currency"}]
iex> Compatibility.breaks(old, old)
[]
iex> Compatibility.breaks(nil, nil)
:error

Each break names its kind first and the field second. The kind is the row of the record's table the break came from - :field_removed, :type_changed, :made_required, :required_added or :made_optional, one per breaking row and no others - so a host words a warning by matching on it rather than by comparing the two declarations again.

Both changes break: dropping authorized_at takes the field away, and relaxing currency takes away the promise that the value is there, which the read check has read as not covered since decision 8 was amended. What does not break is a widening - an optional field added - and a one_of, which is a completion hint and not a constraint however it is edited.

iex> alias StatifierDatamodel.{Compatibility, Declarations}
iex> hinted = fn one_of -> Declarations.from_document(%{"types" => [
...>   %{"name" => "cards.card", "kind" => "record", "label" => "Card",
...>     "fields" => [
...>       %{"name" => "brand", "type" => "string", "required?" => true,
...>         "one_of" => one_of}]}]}) end
iex> {:ok, old} = Declarations.fetch(hinted.(["visa", "mastercard", "amex"]), "cards.card")
iex> {:ok, new} = Declarations.fetch(hinted.(["visa"]), "cards.card")
iex> Compatibility.breaks(old, new)
[]

This answers may a reader keep reading, not did anything change.

Coverage of a map against a shape

StatifierDatamodel.Coverage.missing/3 returns the name of every required field of a shape that a map does not fill, in declaration order. "Fill" is presence of the key with a non-nil value; the value's type is the expression language's question, not this one's. A name that is not declared, or that is a record rather than a shape, is :error.

iex> alias StatifierDatamodel.{Coverage, Declarations}
iex> declarations = Declarations.from_document(%{"types" => [
...>   %{"name" => "Settleable", "kind" => "shape", "label" => "Settleable",
...>     "fields" => [
...>       %{"name" => "amount_cents", "type" => "integer", "required?" => true},
...>       %{"name" => "currency", "type" => "string", "required?" => true},
...>       %{"name" => "authorized_at", "type" => "datetime", "required?" => true}]},
...>   %{"name" => "cards.card", "kind" => "record", "label" => "Card",
...>     "fields" => [
...>       %{"name" => "last4", "type" => "string", "required?" => true}]}]})
iex> Coverage.missing(declarations, "Settleable", %{"amount_cents" => 4200, "currency" => "USD"})
{:ok, ["authorized_at"]}
iex> Coverage.missing(declarations, "Settleable", %{"currency" => nil})
{:ok, ["amount_cents", "currency", "authorized_at"]}
iex> Coverage.missing(declarations, "cards.card", %{})
:error

Value kinds for an expression editor

StatifierDatamodel.Index.path_types/1 projects the index to %{path => kind | {:list, kind} | {:one_of, values}}, in the expression language's own vocabulary: integer and decimal are both :number, a list with a scalar item_type is {:list, kind}, and a drawable one_of wins over the kind. An object, a list with no usable item_type and an unknown type are absent from the map - absence means unknown, not wrong, and an editor handed the map treats a path it does not contain exactly as it treats every path today.

iex> alias StatifierDatamodel.Index
iex> %{"scopes" => [%{"scope" => "local", "entries" => [
...>   %{"path" => "amount_cents", "type" => "integer"},
...>   %{"path" => "risk_reasons", "type" => "list", "item_type" => "string"},
...>   %{"path" => "card", "type" => "object", "fields" => [
...>     %{"path" => "card.brand", "type" => "string",
...>       "one_of" => ["visa", "mastercard", "amex"]},
...>     %{"path" => "card.expires_on", "type" => "date"}]}]}]}
...> |> Index.index()
...> |> Index.path_types()
%{
  "amount_cents" => :number,
  "risk_reasons" => {:list, :string},
  "card.brand" => {:one_of, ["visa", "mastercard", "amex"]},
  "card.expires_on" => :date
}

What is not here

Every function is pure and total over an admitted document, and returns a fact - a set, an index, a boolean, a list of names, a map. Nothing here produces a finding, a severity or a verdict: an undeclared path is unknown, not wrong, and an unsatisfied read is something a consumer assigns a severity to in its own record.

What is deliberately elsewhere: the environment walk over a block document, which needs the block tree and stays in statifier_blocks; anything that renders, which is statifier_ui's; and any runtime enforcement, which no record in the family has asked for.

Who takes this

statifier_blocks reads the document through this package today, for the typed environment's read check. statifier_ui takes path_types/1's map as the expression editor's assign in a later release. Neither takes the other for the document, and this package depends on nothing in the family - that property is the whole reason it exists, and a runtime dependency added here is a decision to record.

License

MIT - see LICENSE.