Summarizer

Differentiability Advantage

The ability to backpropagate through the computation is highlighted as a key difference from external tools, making this a trainable computational substrate.

← Back to Executing programs inside transformers with exponentially faster inference

The debate centers on whether executing code internally within a transformer offers genuine advantages over external tools, with some users questioning if the approach provides measurable improvements in speed or performance. However, proponents argue that the true breakthrough lies in differentiability, which transforms the execution trace into a "trainable substrate" capable of backpropagating gradients directly through the computation. This internal integration creates a powerful systems primitive that could lead to faster decoding and the ability to embed specialized solvers, such as WASM virtual machines, directly into larger models via mixture-of-experts architectures.

2 comments tagged with this topic

View on HN · Topics
This shows the downside of using AI to write up your project. I see the eloquent sentences, but don't get the message. > This works, but the actual execution happened outside the model. The model specified the computation, then waited for an external system to carry it out. > Our transformer also emits a program, but instead of pausing for an external tool, it executes that program itself, step by step, within the same transformer. What's the benefit? Is it speed? Where are the benchmarks? Is it that you can backprop through this computation? Do you do so? Why is it good that it's "inside" the model? Just making it more elegant and nice? The tool was already "inside" the overall hybrid system. What's the actual problem?
View on HN · Topics
> Is it speed? > Is it that you can backprop through this computation? Do you do so? With respect, I feel that you may not have read the article. > Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model. and, > By storing points across nested convex hulls, this yields a decoding cost of O(k+log⁡ n). and, > Regardless of their eventual capability ceiling, they already suggest a powerful systems primitive for speeding up larger models. So yes, and yes. > Where are the benchmarks? Not clear what they should benchmark it against. They do compare speed to a normal KV Cache. As for performance.. if it's actually executing a Sudoku solver with a 100% success rate, it seems pretty trivial to find any model doing < 100% success rate. Sure, it would be nice to see the data here, agree with you there. Personally I think it would be really interesting to see if this method can be combined with a normal model MoE-style. It is likely possible, the router module should pick up quite quickly that it predicts the right tokens for some subset of problems deterministically. I like the idea of embed all sorts of general solvers directly into the model, like a prolog solver for example. In fact it never would have occurred to me to just go straight for WASM, pretty interesting choice to directly embed a VM. But it makes me wonder what "smaller" interpreters could be useful in this context.