Tweet by Marcello Herreshoff:

Your barrel transcript is an example of a specific LLM pathology I call "template fixation". That's when, if certain features of the input remind the LLM enough of some pattern that occurs often in the training data, the LLM aggressively applies that pattern, even if it doesn't…