MoE Integration Possibilities

Speculation about combining this approach with Mixture of Experts architectures, where routers could select deterministic solvers for appropriate problem subsets.

The discussed approach introduces a differentiable computational substrate that allows models to backpropagate directly through execution traces, offering a high-speed alternative to traditional external tools. By integrating these deterministic solvers as "experts" within a Mixture of Experts (MoE) framework, a model’s router could learn to delegate specific problem subsets to reliable algorithms for perfect accuracy. This architecture opens the door to embedding entire virtual machines or specialized interpreters, such as Prolog, directly into the model’s fabric rather than relying on external calls. Ultimately, these primitives suggest a future where large models are enhanced by internal, trainable logic modules that bridge the gap between neural intuition and algorithmic precision.

View on HN · Topics

> Is it speed?

> Is it that you can backprop through this computation? Do you do so?

With respect, I feel that you may not have read the article.

> Because the execution trace is part of the forward pass, the whole process remains differentiable: we can even propagate gradients through the computation itself. That makes this fundamentally different from an external tool. It becomes a trainable computational substrate that can be integrated directly into a larger model.

and,

> By storing points across nested convex hulls, this yields a decoding cost of O(k+log⁡ n).

and,

> Regardless of their eventual capability ceiling, they already suggest a powerful systems primitive for speeding up larger models.

So yes, and yes.

> Where are the benchmarks?

Not clear what they should benchmark it against. They do compare speed to a normal KV Cache. As for performance.. if it's actually executing a Sudoku solver with a 100% success rate, it seems pretty trivial to find any model doing < 100% success rate. Sure, it would be nice to see the data here, agree with you there.

Personally I think it would be really interesting to see if this method can be combined with a normal model MoE-style. It is likely possible, the router module should pick up quite quickly that it predicts the right tokens for some subset of problems deterministically. I like the idea of embed all sorts of general solvers directly into the model, like a prolog solver for example. In fact it never would have occurred to me to just go straight for WASM, pretty interesting choice to directly embed a VM. But it makes me wonder what "smaller" interpreters could be useful in this context.

Summarizer