Tweet by Mehrdad Farajtabar:

5/ #Result 2: The fragility of supposed LLM reasoning. LLMs remain sensitive to changes in proper names (e.g., people, foods, objects), and even more so when numbers are altered. Would a grade-school student's math test score vary by ~10% if we only changed the names? pic.twitter.com/mvZUaRR8DB