TensorFlow Lite uses a "delegate" system to hand off parts of a model graph to specialized backends — GPU delegates, NPU delegates, and, for CPU execution, the XNNPACK delegate. Understanding this handoff clears up a lot of confusion about where the actual computation happens.
What a delegate does
When TFLite loads a model, it examines the graph and decides which operators a given delegate can handle. Supported operators get replaced with calls into that delegate; anything unsupported still runs through TFLite's built-in reference kernels. The XNNPACK delegate covers a large share of common operators — convolutions, pooling, activations, fully connected layers — which is why it's enabled by default in current TFLite builds for floating-point models.
The handoff, step by step
- TFLite parses the
.tflitemodel file into an internal graph representation - It walks the graph, checking which nodes the XNNPACK delegate can execute
- Supported subgraphs are partitioned out and replaced with a single delegate node
- At inference time, that delegate node calls into XNNPACK's subgraph API, which runs the actual operators using its optimized kernels
- Results flow back into the rest of the TFLite graph for any remaining unsupported operators
Checking whether it's active
Because the delegate is applied automatically, it's easy to assume XNNPACK is doing the work without confirming it. TFLite's verbose or profiling logs typically report which nodes were delegated versus run through the reference path — checking that output is the most reliable way to confirm XNNPACK is actually in the hot path for your specific model.
When operators fall back
Not every custom or unusual operator has an XNNPACK-backed implementation. When a model includes one, TFLite quietly falls back to its slower reference kernel for just that node, which can create a surprising performance cliff in an otherwise fast model. If profiling shows unexpectedly slow inference, checking for one or two non-delegated operators breaking up an otherwise delegated graph is a good first diagnostic step.
Quantized models
The delegate also covers INT8-quantized operators, not just floating-point ones, which matters for mobile deployments where quantization is used specifically to shrink model size and improve speed. The quantized kernel paths are a separate code path within XNNPACK from the FP32 ones covered elsewhere on this site.
Related reading
For the ARM-specific kernel work underlying this on phones, see how XNNPACK optimizes for ARM NEON. For general speed tuning once the delegate is confirmed active, see cutting mobile inference latency.