TFLite-Elixir

TensorFlow Lite-Elixir binding with TPU support.

OS	Arch	ABI	Build Status	Has Precompiled Library
Ubuntu 20.04	x86_64	gnu		Yes
Ubuntu 20.04	arm64	gnu		Yes
Ubuntu 20.04	armv7l	gnueabihf		Yes
Ubuntu 20.04	riscv64	gnu		Yes
macOS 11 Big Sur	x86_64	darwin		Yes
macOS 11 Big Sur	arm64	darwin		Yes

Try it in Livebook

# will download and install precompiled version
Mix.install([
  {:tflite_elixir, "~> 0.1.0", github: "cocoa-xu/tflite_elixir"}
])

# test data can be found in the test directory
interpreter = TFLiteElixir.Interpreter.new!("test/test_data/mobilenet_v2_1.0_224_inat_bird_quant.tflite")
input = 
  StbImage.read_file!("test/test_data/parrot.jpeg")
  |> StbImage.resize(224, 224)
  |> StbImage.to_nx()
output = TFLiteElixir.Interpreter.predict(interpreter, input)
  |> TFLiteElixir.TFLiteTensor.to_nx(Nx.BinaryBackend)

Some livebook examples can be found in the examples directory.

Nerves Support

1. If prefer precompiled binaries

# for example
export MIX_TARGET=rpi4

# There is no need to explicitly set CPU architecture 
#   for the precompiled libedgetpu binaries. The arch
#   is automatically detected by the `TARGET_ARCH`,
#   `TARGET_OS` and `TARGET_ABI` environment vars.
#
# However, if you are using your own nerves target
#   you can manually set the correct arch, e.g., 
#   set `aarch64` for rpi4.
#
# Possible values including
# - aarch64
# - armv7l
# - riscv64
# - x86_64
export TFLITE_ELIXIR_CORAL_LIBEDGETPU_LIBRARIES=aarch64

1. If prefer not to use precompiled binaries

# for example
export MIX_TARGET=rpi4
# then set env var TFLITE_ELIXIR_PREFER_PRECOMPILED to NO
export TFLITE_ELIXIR_PREFER_PRECOMPILED=NO

Demo

Mix Task Demo

1. List all available Edge TPU

mix list_edgetpu

1. Image classification

mix help classify_image

# Note: The first inference on Edge TPU is slow because it includes,
# loading the model into Edge TPU memory
mix classify_image \
  --model test/test_data/mobilenet_v2_1.0_224_inat_bird_quant.tflite \
  --input test/test_data/parrot.jpeg \
  --labels test/test_data/inat_bird_labels.txt

Output from the mix task

----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes, loading the model into Edge TPU memory.
6.7ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.70703

1. Object detection

mix help detect_image

# Note: The first inference on Edge TPU is slow because it includes,
# loading the model into Edge TPU memory
mix detect_image \
  --model test/test_data/ssd_mobilenet_v2_coco_quant_postprocess.tflite \
  --input test/test_data/cat.jpeg \
  --labels test/test_data/coco_labels.txt

Output from the mix task

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
----INFERENCE TIME----
13.2ms
cat
  id   : 16
  score: 0.953
  bbox : [3, -1, 294, 240]

test files used here are downloaded from google-coral/test_data and wikipedia.

Demo code

Model: mobilenet_v2_1.0_224_inat_bird_quant.tflite

Input image:

• parrot.jpg
• Or use pre-converted input parrot.bin

Labels: inat_bird_labels.txt

alias Evision, as: Cv
alias TFLiteElixir, as: TFLite

# load labels
labels = File.read!("inat_bird_labels.txt") |> String.split("\n")

# load tflite model
filename = "mobilenet_v2_1.0_224_inat_bird_quant.tflite"
model = TFLite.FlatBufferModel.buildFromFile!(filename)
resolver = TFLite.Ops.Builtin.BuiltinResolver.new!()
builder = TFLite.InterpreterBuilder.new!(model, resolver)
interpreter = TFLite.Interpreter.new!()
:ok = TFLite.InterpreterBuilder.build!(builder, interpreter)
:ok = TFLite.Interpreter.allocateTensors!(interpreter)

# verify loaded model, feel free to skip
# [0] = TFLite.Interpreter.inputs!(interpreter)
# [171] = TFLite.Interpreter.outputs!(interpreter)
# "map/TensorArrayStack/TensorArrayGatherV3" = TFLite.Interpreter.getInputName!(interpreter, 0)
# "prediction" = TFLite.Interpreter.getOutputName!(interpreter, 0)
# input_tensor = TFLite.Interpreter.tensor!(interpreter, 0)
# [1, 224, 224, 3] = TFLite.TFLiteTensor.dims!(input_tensor)
# {:u, 8} = TFLite.TFLiteTensor.type(input_tensor)
# output_tensor = TFLite.Interpreter.tensor!(interpreter, 171)
# [1, 965] = TFLite.TFLiteTensor.dims!(output_tensor)
# {:u, 8} = TFLite.TFLiteTensor.type(output_tensor)

# parrot.bin - if you don't have :evision
binary = File.read!("parrot.bin")
# parrot.jpg - if you have :evision
# load image, resize it, covert to RGB and to binary 
binary = 
  Cv.imread("parrot.jpg")
  |> Cv.resize({224, 224})
  |> Cv.cvtColor(Cv.cv_COLOR_BGR2RGB)
  |> Cv.Mat.to_binary(mat)

# set input, run forwarding, get output
TFLite.Interpreter.input_tensor(interpreter, 0, binary)
TFLite.Interpreter.invoke(interpreter)
output_data = TFLite.Interpreter.output_tensor!(interpreter, 0)

# if you have :nx
# get predicted label
output_data
|> Nx.from_binary(:u8)
|> Nx.argmax()
|> Nx.to_scalar()
|> then(&Enum.at(labels, &1))

Coral Support

Dependencies

For macOS

# only required if not using precompiled binaries
# for compiling libusb
brew install autoconf automake

For some Linux OSes you need to manually execute the following command to update udev rules, otherwise, libedgetpu will fail to initialize Coral devices.

mix deps.get
bash "3rd_party/cache/${TFLITE_ELIXIR_CORAL_LIBEDGETPU_RUNTIME}/edgetpu_runtime/install.sh"

Compile-Time Environment Variable

• TFLITE_ELIXIR_CORAL_SUPPORT

  Enable Coral Support.

  Default to YES.

• TFLITE_ELIXIR_CORAL_USB_THROTTLE

  Throttling USB Coral Devices. Please see the official warning here, google-coral/libedgetpu.

  Default value is YES.

  Note that only when TFLITE_ELIXIR_CORAL_USB_THROTTLE is set to NO, :tflite_elixir will use the non-throttled libedgetpu libraries.

• TFLITE_ELIXIR_CORAL_LIBEDGETPU_LIBRARIES

  Choose which ones of the libedgetpu libraries to copy to the priv directory of the :tflite_elixir app.

  Default value is native - only native libraries will be downloaded and copied. native corresponds to the host OS and CPU architecture when compiling this library.

  When set to a specific value, e.g, darwin_arm64 or darwin_x86_64, then the corresponding one will be downloaded and copied. This option is expected to be used for cross-compiling, like with nerves.

  Available values for this option are:

  Value	OS/CPU
  aarch64	Linux arm64
  armv7l	Linux armv7
  k8	Linux x86_64
  x86_64	Linux x86_64
  riscv64	Linux riscv64
  darwin_arm64	macOS Apple Silicon
  darwin_x86_64	macOS x86_64
  x64_windows	Windows x86_64

• TFLITE_ELIXIR_CACHE_DIR

  Cache directory for the runtime zip file.

  Default value is ./3rd_party/cache.

Installation

If available in Hex, the package can be installed by adding tflite_elixir to your list of dependencies in mix.exs:

def deps do
  [
    {:tflite_elixir, "~> 0.1.0", github: "cocoa-xu/tflite_elixir"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/tflite_elixir.