nanoGPT Arithmetic Explorer

Live Script that trains a small MATLAB GPT from scratch to add integers in any base.

The Live Script nanoGPT Explorer [1] based on Karpthy's template [2] introduces a toy GPT trained on Shakespeare and novels to illustrate how large language models work. The model attempts to learn English grammar and a large vocabulary and produces a few words of babble - remarkable but not mind-blowing. Arithmetic (addition specifically) poses a problem more appropriate for a tiny GPT. It is about “getting” a simple algorithm. (Here: Start by adding columns on the right, learn to “carry” left, then chain operations.) Once a model gets it ("groks”), it gets perfect verifiable results.

This Live Script walks a small transformer through five experiments that explore difficulties and ways to remove them. The model, the training loop, and the evaluation are all in the engine files shipped with this script (arithGPT.m for a single base and arithGPT_anybase.m for mixed-base experiments) and run on a CPU; nothing calls a cloud service or a GPU.

Five acts trace an experimental path: (1) random training that can stall on the carry even with some standard tricks; (2) a difficulty curriculum that explores structured not random training; (3) a scratchpad that writes the carries out loud during random training; (4) both together; and (5) one model trained on several bases at once, tested on a base it never saw.

The training execution time for each experiment is of order 10-30 minutes on a 2021 MacBook with an M1 chip. By default every act displays a pre-computed figure. Each act also has a "Try this" switch that retrains that experiment live with your own settings and saves results in a folder. "Challenges" at the end suggest experiments coding extensions. A background information section in the Live Script frames Live Scrpt goals and use. An accompanying text document details how to use the engine functions for other studies. This script will interest students and instructors of machine learning and computational physics. Additional educational Live Scripts by the author are here.