TFLiteBEAM

TensorFlow Lite BEAM bindings with optional EdgeTPU support.

Hex.pmCoverage Status

OSArchABIBuild StatusHas Precompiled Library
Ubuntu 20.04x86_64gnuCIYes
Ubuntu 20.04arm64gnuCIYes
Ubuntu 20.04armv7lgnueabihfCIYes
Ubuntu 20.04armv6gnueabihfCIYes
Ubuntu 20.04riscv64gnuCIYes
macOS 15 Sequoiax86_64darwinCIYes
macOS 14 Sonomaarm64darwinCIYes

Delegates

tflite_beam_interpreter_builder:build/2 attaches an XNNPACK delegate for you, unless you have attached one yourself. TfLite would otherwise apply XNNPACK on its own, invisibly, inside allocate_tensors/1 -- with a thread count nothing could reach and no way to decline it. The acceleration is the same; where it happens is now visible, and set_num_threads/2 still reaches it.

{ok, Resolver} = tflite_beam_ops_builtin_builtin_resolver:new(),

{ok, Builder} = tflite_beam_interpreter_builder:new(Model, Resolver),



%% your own delegate instead of the default one

{ok, Delegate} = tflite_beam_delegate:xnnpack(#{num_threads => 4}),

ok = tflite_beam_interpreter_builder:add_delegate(Builder, Delegate),



ok = tflite_beam_interpreter_builder:build(Builder, Interpreter).

tflite_beam_delegate:available/0 lists the delegate kinds this build can create: XNNPACK on every target except armv6 and armv7l, where nothing is attached and inference runs as it always has.

To go back to TfLite delegating by itself, ask the resolver for it:

{ok, Resolver} = tflite_beam_ops_builtin_builtin_resolver:new(#{apply_default_delegates => true}),

A delegate must outlive every interpreter built from the builder it was added to, so there is no way to detach or free one: the builder and each interpreter hold it for as long as they need it, and it goes when they do.

Delegates from a shared library

Anything implementing TfLite's delegate plugin interface -- tflite_plugin_create_delegate and tflite_plugin_destroy_delegate -- can be loaded at runtime, which covers Edge TPU, a GPU delegate built elsewhere, and vendor delegates this library knows nothing about:

{ok, Delegate} = tflite_beam_delegate:external("/opt/lib/libvendor_delegate.so",

                                               #{device => 0, precision => fp16}),

ok = tflite_beam_interpreter_builder:add_delegate(Builder, Delegate).

Options are handed to the plugin as strings, which is the whole of that ABI, so atoms and integers are converted and at most 256 pairs fit. What the keys mean is the plugin's business. The path is resolved to an absolute one before loading, because the loader is asked for exactly the file named -- a bare libfoo.so would otherwise be searched for wherever the system looks, which is rarely where anyone means. The library is not unloaded afterwards.

Threading

An interpreter, and any delegate attached to it, belongs to one process at a time. TfLite documents tflite::Interpreter as not thread-safe and leaves serialising access to the caller, and nothing here adds a lock of its own -- invoke/1 runs on a dirty scheduler, so two processes sharing one interpreter really do run it on two OS threads at once. Delegates are the same: nothing documents a TfLiteDelegate as safe to back two interpreters simultaneously, and XNNPACK's demonstrably is not.

Coral Support

libedgetpu is itself a TfLite delegate plugin, so an Edge TPU can be attached like any other delegate -- which means it composes with set_num_threads/2 and with whatever else is on the builder:

{ok, Delegate} = tflite_beam_coral:edge_tpu_delegate(),

ok = tflite_beam_interpreter_builder:add_delegate(Builder, Delegate),

ok = tflite_beam_interpreter_builder:build(Builder, Interpreter).

tflite_beam_coral:make_edge_tpu_interpreter/2 still works and is unchanged. It builds its own interpreter internally, though, so nothing set on a builder reaches it; the delegate above is the composable route. Asking for a device that is not there is an ordinary {error, Reason} from edge_tpu_delegate/1.

Both routes have been checked to produce identical output on a USB Coral accelerator, running mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite against libedgetpu 0.1.14 on macOS arm64.

Dependencies

For macOS

# only required if not using precompiled binaries

# for compiling libusb

brew install autoconf automake

For some Linux OSes you need to manually execute the following command to update udev rules, otherwise, libedgetpu will fail to initialize Coral devices.

bash "3rd_party/cache/${TFLITE_BEAM_CORAL_LIBEDGETPU_RUNTIME}/edgetpu_runtime/install.sh"

Compile-Time Environment Variable

Installation

Add tflite_beam to your list of dependencies in rebar.config:

{deps, [

  {tflite_beam, "0.3.12"}

]}

Documentation is published on HexDocs.

Tests

rebar3 ct

The model fixtures live in test/models/, so the suite needs no network and runs against a precompiled install as well as a build from source.

Upstream Dependencies