TFLiteElixir

TensorFlow Lite Elixir bindings with optional EdgeTPU support.

For pure Erlang bindings, please see cocoa-xu/tflite_beam.

Hex.pm

Getting Started

Run in Livebook

A general workflow looks like this,

# will download and install precompiled version

Mix.install([

  {:tflite_elixir, "~> 0.3"}

])



# parrot.jpeg and the tflite file can be found in the test/test_data directory

interpreter = TFLiteElixir.Interpreter.new!("/path/to/mobilenet_v2_1.0_224_inat_bird_quant.tflite")

input =

  StbImage.read_file!("/path/to/parrot.jpeg")

  |> StbImage.resize(224, 224)

  |> StbImage.to_nx()



[output_tensor_0] = TFLiteElixir.Interpreter.predict(interpreter, input)

indices_nx = Nx.flatten(output_tensor_0)



# get top k predictions (numerical id of the class)

# classes can be found in this file,

# https://raw.githubusercontent.com/cocoa-xu/tflite_elixir/main/test/test_data/inat_bird_labels.txt

# each line corresponds to a class

# and the first line = id 0

top_k = 5

sorted_indices = Nx.argsort(indices_nx, direction: :desc)

top_k_indices = Nx.take(sorted_indices, Nx.iota({top_k}))

top_k_preds = Nx.to_flat_list(top_k_indices)

And there is an experimental ImageClassification module that does everything for you. It supports both CPU and TPU, and it will show more information, including scores (confidence) and the class name of the predicted results. It's also more flexible where you can adjust different parameters like top_k and threshold (for confidence) and etc.

iex> alias TFLiteElixir.ImageClassification

iex> {:ok, pid} = ImageClassification.start("/path/to/mobilenet_v2_1.0_224_inat_bird_quant.tflite")

iex> ImageClassification.predict(pid, "/path/to/parrot.jpeg")

%{class_id: 923, score: 0.70703125}

iex> ImageClassification.set_label_from_associated_file(pid, "inat_bird_labels.txt")

:ok

iex> ImageClassification.predict(pid, "/path/to/parrot.jpeg")

%{class_id: 923, label: "Ara macao (Scarlet Macaw)", score: 0.70703125}

iex> ImageClassification.predict(pid, "/path/to/parrot.jpeg", top_k: 3)

[

  %{class_id: 923, label: "Ara macao (Scarlet Macaw)", score: 0.70703125},

  %{

    class_id: 837,

    label: "Platycercus elegans (Crimson Rosella)",

    score: 0.078125

  },

  %{

    class_id: 245,

    label: "Coracias caudatus (Lilac-breasted Roller)",

    score: 0.01953125

  }

]

There is an experimental ObjectDetection module in the same shape. It also supports both CPU and TPU, and returns one entry per detection with its class id, score, label and bounding box, [ymin, xmin, ymax, xmax], in the coordinates of the image you gave it.

iex> alias TFLiteElixir.ObjectDetection

iex> {:ok, pid} = ObjectDetection.start("/path/to/ssd_mobilenet_v2_coco_quant_postprocess.tflite")

iex> ObjectDetection.predict(pid, "/path/to/cat.jpeg")

[%{class_id: 16, label: nil, score: 0.93359375, bbox: [3, -1, 294, 240]}]

iex> ObjectDetection.set_label(pid, "/path/to/coco_labels.txt")

:ok

iex> ObjectDetection.predict(pid, "/path/to/cat.jpeg")

[%{class_id: 16, label: "cat", score: 0.93359375, bbox: [3, -1, 294, 240]}]

iex> ObjectDetection.predict(pid, "/path/to/cat.jpeg", threshold: 0.99)

[]

To run it on a Coral device, start it with use_tpu: true and an Edge TPU model.

Signatures

A model exported with signatures names its inputs and outputs, so neither side has to depend on the order the tensors happen to be listed in.

iex> alias TFLiteElixir.{Interpreter, SignatureRunner}

iex> {:ok, runner} = Interpreter.get_signature_runner(interpreter, "serving_default")

iex> SignatureRunner.input_names!(runner)

["input_1"]

iex> {:ok, outputs} = SignatureRunner.predict(runner, %{"input_1" => input_data})

iex> Map.keys(outputs)

["output_1"]

Passing nil instead of a key asks for the primary subgraph, which also works for models that declare no signatures at all.

Nerves Support

Prebuilt firmware (Experimental)

Nerves

Prebuilt firmwares are available here. Nightly builds can be found here.

Select the most recent run and scroll down to the Artifacts section, download the firmware file for your board and run

fwup /path/to/the/downloaded/firmware.fw

In the nerves build, tflite_elixir is integrated as one of the dependencies of the nerves_livebook project. This means that you can use livebook (as well as other pre-pulled libraries) to explore and evaluate the tflite_elixir project.

The default password of the livebook is nerves (as the time of writing, if it does not work, please check the nerves_livebook project).

Build from Source

  1. If prefer precompiled binaries
# for example

export MIX_TARGET=rpi4



# There is no need to explicitly set CPU architecture

#   for the precompiled libedgetpu binaries. The arch

#   is automatically detected by the `TARGET_ARCH`,

#   `TARGET_OS` and `TARGET_ABI` environment vars.

#

# However, if you are using your own nerves target

#   you can manually set the correct arch, e.g.,

#   set `aarch64` for rpi4.

#

# Possible values including

# - aarch64

# - armv7l

# - armv6

# - riscv64

# - x86_64

export TFLITE_BEAM_CORAL_LIBEDGETPU_LIBRARIES=aarch64
  1. If prefer not to use precompiled binaries
# for example

export MIX_TARGET=rpi4

# then set env var TFLITE_BEAM_PREFER_PRECOMPILED to false

export TFLITE_BEAM_PREFER_PRECOMPILED=false

Demo

Mix Task Demo

  1. List all available Edge TPU
mix list_edgetpu
  1. Image classification
mix help classify_image



# Note: The first inference on Edge TPU is slow because it includes,

# loading the model into Edge TPU memory

mix classify_image \

  --model test/test_data/mobilenet_v2_1.0_224_inat_bird_quant.tflite \

  --input test/test_data/parrot.jpeg \

  --labels test/test_data/inat_bird_labels.txt

Output from the mix task

----INFERENCE TIME----

Note: The first inference on Edge TPU is slow because it includes, loading the model into Edge TPU memory.

6.7ms

-------RESULTS--------

Ara macao (Scarlet Macaw): 0.70703
  1. Object detection
mix help detect_image



# Note: The first inference on Edge TPU is slow because it includes,

# loading the model into Edge TPU memory

mix detect_image \

  --model test/test_data/ssd_mobilenet_v2_coco_quant_postprocess.tflite \

  --input test/test_data/cat.jpeg \

  --labels test/test_data/coco_labels.txt

Output from the mix task

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

----INFERENCE TIME----

13.2ms

cat

  id   : 16

  score: 0.953

  bbox : [3, -1, 294, 240]

test files used here are downloaded from google-coral/test_data and wikipedia.

Demo code

Model: mobilenet_v2_1.0_224_inat_bird_quant.tflite

Input image:

Labels: inat_bird_labels.txt

alias Evision, as: Cv

alias TFLiteElixir, as: TFLite



# load labels

labels = File.read!("inat_bird_labels.txt") |> String.split("\n")



# load tflite model

filename = "mobilenet_v2_1.0_224_inat_bird_quant.tflite"

model = TFLite.FlatBufferModel.build_from_file(filename)

resolver = TFLite.Ops.Builtin.BuiltinResolver.new!()

builder = TFLite.InterpreterBuilder.new!(model, resolver)

interpreter = TFLite.Interpreter.new!()

:ok = TFLite.InterpreterBuilder.build!(builder, interpreter)

:ok = TFLite.Interpreter.allocate_tensors(interpreter)



# verify loaded model, feel free to skip

# [0] = TFLite.Interpreter.inputs!(interpreter)

# [171] = TFLite.Interpreter.outputs!(interpreter)

# "map/TensorArrayStack/TensorArrayGatherV3" = TFLite.Interpreter.get_input_name!(interpreter, 0)

# "prediction" = TFLite.Interpreter.get_output_name!(interpreter, 0)

# input_tensor = TFLite.Interpreter.tensor(interpreter, 0)

# [1, 224, 224, 3] = TFLite.TFLiteTensor.dims(input_tensor)

# {:u, 8} = TFLite.TFLiteTensor.type(input_tensor)

# output_tensor = TFLite.Interpreter.tensor(interpreter, 171)

# [1, 965] = TFLite.TFLiteTensor.dims(output_tensor)

# {:u, 8} = TFLite.TFLiteTensor.type(output_tensor)



# parrot.bin - if you don't have :evision

binary = File.read!("parrot.bin")

# parrot.jpg - if you have :evision

# load image, resize it, covert to RGB and to binary

binary =

  Cv.imread("parrot.jpg")

  |> Cv.resize({224, 224})

  |> Cv.cvtColor(Cv.cv_COLOR_BGR2RGB)

  |> Cv.Mat.to_binary(mat)



# set input, run forwarding, get output

TFLite.Interpreter.input_tensor(interpreter, 0, binary)

TFLite.Interpreter.invoke(interpreter)

output_data = TFLite.Interpreter.output_tensor!(interpreter, 0)



# if you have :nx

# get predicted label

output_data

|> Nx.from_binary(:u8)

|> Nx.argmax()

|> Nx.to_scalar()

|> then(&Enum.at(labels, &1))

Delegates

TFLiteElixir.InterpreterBuilder.build/2 attaches an XNNPACK delegate for you, unless you have attached one yourself. TfLite would otherwise apply XNNPACK on its own, invisibly, inside allocate_tensors/1 -- with a thread count nothing could reach and no way to decline it. The acceleration is the same; where it happens is now visible.

alias TFLiteElixir.{Delegate, InterpreterBuilder}



{:ok, delegate} = Delegate.xnnpack(num_threads: 4)

:ok = InterpreterBuilder.add_delegate!(builder, delegate)

Delegate.available/0 lists what this build can create: :xnnpack on every target except armv6 and armv7l, and :external everywhere.

To hand delegation back to TfLite, ask the resolver for it:

resolver = TFLiteElixir.Ops.Builtin.BuiltinResolver.new!(apply_default_delegates: true)

Anything implementing TfLite's delegate plugin interface can be loaded at runtime, which covers a GPU delegate built elsewhere and any vendor delegate:

{:ok, delegate} =

  Delegate.external("/opt/lib/libvendor_delegate.so", device: 0, precision: :fp16)

A delegate must outlive every interpreter built from the builder it was added to, so there is no way to detach or free one: the builder and each interpreter hold it for as long as they need it. An interpreter, and any delegate attached to it, belongs to one process at a time.

Coral Support

An Edge TPU is reachable as a delegate too, which puts it on the ordinary builder path -- composable with set_num_threads/2 and with anything else attached:

{:ok, delegate} = TFLiteElixir.Coral.edge_tpu_delegate()

:ok = TFLiteElixir.InterpreterBuilder.add_delegate!(builder, delegate)

TFLiteElixir.Coral.make_edge_tpu_interpreter/2 still works and is unchanged. It builds its own interpreter internally, though, so nothing set on a builder reaches it. Both routes produce identical output; asking for a device that is not attached is an ordinary {:error, reason}.

Dependencies

For macOS

# only required if not using precompiled binaries

# for compiling libusb

brew install autoconf automake

For some Linux OSes you need to manually execute the following command to update udev rules, otherwise, libedgetpu will fail to initialize Coral devices.

mix deps.get

bash "3rd_party/cache/${TFLITE_BEAM_CORAL_LIBEDGETPU_RUNTIME}/edgetpu_runtime/install.sh"

Compile-Time Environment Variable

Installation

Add :tflite_elixir to your list of dependencies in mix.exs:

def deps do

  [

    {:tflite_elixir, "~> 0.3"}

  ]

end

The 1.0.0 release candidates build the runtime from LiteRT rather than from TensorFlow. Hex never resolves a pre-release from a range, so name it exactly:

def deps do

  [

    {:tflite_elixir, "1.0.0-rc2"}

  ]

end

~> 0.3 will not reach it, which is deliberate: a two part 0.x requirement means everything below 1.0.0, so releasing this as 0.4.0 would have moved every existing user onto a different upstream without their asking. Nothing in the Elixir API was removed or renamed. The one answer that changed is TFLiteElixir.tflite_version/0, which reports LiteRT's version rather than TensorFlow's; the two version lines are not comparable, so LiteRT's 2.2.0 is newer than TensorFlow's 2.21.0 rather than older. That number is what a delegate plugin has to match.

This release requires nx ~> 0.11, up from ~> 0.5. LiteRT reports two eight bit float formats, and Nx gained {:f, 8} for E5M2 in 0.9.0 and {:f8_e4m3fn, 8} for E4M3FN in 0.11.0. On an older Nx, TFLiteTensor.to_nx/2 raises ArgumentError on either of those tensors, so the floor states what the binding can actually hand over rather than leaving it to be discovered.

What the numbers actually say

Measured on an Apple M4 Max, macOS 15.7.3, with mobilenet_v2_1.0_224_inat_bird_quant, 64 images per run. Your machine will give different figures; what is worth carrying over is the shape of them.

Batching buys almost nothing.resize_input_tensor/3 to a batch and feeding several images at once is a real thing you can do, and it barely pays:

batchCPU, per imageEdge TPU, per image
17.50 ms10.01 ms
47.19 ms9.89 ms
87.23 ms9.82 ms
167.25 ms9.78 ms

Sixteen times the batch is 3% off the per image cost on CPU and 2% on the TPU. TFLite's CPU kernels already thread inside a single image, so the batch axis is an outer loop rather than the thing that fills the machine; on the USB TPU the bytes crossing the wire scale with the batch, so batching them together saves no transfer. Batch if it suits how your inputs arrive, not to go faster.

Two of the three detection models here cannot batch at all.ssd_mobilenet_v2_coco_quant_postprocess fails allocate_tensors/1 after the resize. lite-model_efficientdet_lite4 returns :ok and then leaves its output at {1, 25, 4}: the batch never reaches the answer, so four images in gets one result out and the other three are quietly lost. Detection models with fixed output postprocessing generally behave this way. If you batch, check the output shape followed your resize before trusting it.

The Edge TPU is not automatically the fast path. On this machine the USB Coral is 33% slower than the CPU for the same model, because a strong CPU plus XNNPACK beats a 150KB round trip over USB. On a Raspberry Pi that ordering reverses and it is not close. Measure on the machine you will deploy to.

Documentation can be found at https://hexdocs.pm/tflite_elixir.