According to @karpathy, enabling FP8 training in the nanochat GPT-2 reproduction delivered a 4.3 percent improvement in time to GPT-2, reducing training to 2.91 hours on a single 8x H100 node. According to @karpathy, at spot pricing an 8x H100 run can cost about 20 dollars, while a previous 3.04 hour run cost about 73 dollars, highlighting roughly a 600 times cost reduction versus OpenAI’s original GPT-2 training. According to @karpathy, FP8 on H100 offers 2 times theoretical FLOPs but practical gains are limited by scaling conversion overhead, partial non compute bound training, and small GEMMs at GPT-2 scale, yielding about 7.3 percent per step speedup and roughly 5 percent net after adjusting the training horizon. According to @karpathy, torchao reported a 25 percent FP8 speedup on Llama3 8B, implying larger models may benefit more, and he expects further gains by selectively applying FP8 to layers and tightening numerics. According to @karpathy, additional wins came from Flash Attention 3, the Muon optimizer, gated residual and skip connections, and value embeddings, and he published a reproducible setup and a time to GPT-2 leaderboard on GitHub.

According to @karpathy, nanochat can now train a GPT-2 grade large language model for about 73 dollars in roughly 3 hours on a single 8x H100 node, setting a concrete cost and time benchmark for compact LLM training (source: @karpathy). According to @karpathy, GPT-2 remains a favored milestone because it represents the first recognizably modern LLM stack, and his update highlights reproducible, low-cost training of GPT-2 grade models on current-generation GPUs (source: @karpathy).

According to @karpathy, nanochat’s first public miniseries v1 demonstrates compute-optimal LLM training across model sizes at fixed FLOPs with an end-to-end pipeline and reproducible scripts. source: @karpathy on X Jan 7, 2026; nanochat GitHub discussion #420 He reports nanochat reproduces Chinchilla-like scaling with equal exponents on parameters and data near 0.5 and a single compute-independent constant of about 8 tokens per parameter versus 20 reported in Chinchilla. source: @karpathy on X Jan 7, 2026; Hoffmann et al. 2022 Chinchilla The sweep from d10 to d20 achieves non-intersecting training curves at batch sizes around 2^19 (about 0.5M) on one 8x H100 node without gradient accumulation. source: @karpathy on X Jan 7, 2026 He aligns nanochat with GPT-2 and estimated GPT-3 using the CORE score for an objective cross-series comparison. source: @karpathy on X Jan 7, 2026; DCLM paper (CORE score) The total experiment cost is about $100 for roughly 4 hours on 8x H100, with all tuning and code pushed to master for reproduction via scaling_laws.sh and miniseries.sh. source: @karpathy on X Jan 7, 2026; nanochat GitHub discussion #420 This implies roughly $3.1 per H100 GPU-hour for the described run, offering a live reference for pricing compute in AI workloads. source: calculation based on @karpathy on X Jan 7, 2026 For crypto markets, decentralized GPU networks that price or facilitate GPU time make these cost and scaling benchmarks directly relevant to workload pricing and benchmarking on networks like Render Network (RNDR) and Akash Network (AKT). source: Render Network documentation; Akash Network documentation