According to Andrej Karpathy on X, he is running a larger model on NanoChat backed by 8x H100 GPUs and plans to keep the benchmark running for a while, indicating a focus on sustained, production-grade inference performance and scaling behavior (source: Andrej Karpathy). As reported by Karpathy, the setup highlights multi-GPU inference for larger models, a key requirement for low-latency, high-throughput chat workloads and real-time serving (source: Andrej Karpathy). According to Karpathy, this configuration signals opportunities for enterprises to evaluate tokenizer throughput, context window costs, and tensor parallel scaling on H100 clusters for customer support bots and code assistants (source: Andrej Karpathy). As reported by Karpathy, developers can benchmark token-per-second, batch sizing, and KV cache strategies to reduce serving cost per 1K tokens, informing capacity planning on 8x H100 nodes (source: Andrej Karpathy).

According to Andrej Karpathy on Twitter, he is running a larger model on an 8×H100 setup in production for NanoChat and plans to leave the job running for an extended period. As reported by Karpathy’s post, this highlights a production-scale inference workload using NVIDIA H100 GPUs, indicating sustained high-throughput serving and stability testing for a bigger model. According to Karpathy, the configuration suggests enterprises can validate latency, throughput, and cost curves for large model deployments on H100 clusters, informing capacity planning, autoscaling, and GPU utilization strategies. As reported by the Twitter post, this scenario underscores business opportunities in model serving optimization, including quantization, tensor parallelism, and memory-efficient batching to maximize H100 occupancy.

According to Andrej Karpathy on X, NanoChat now trains a GPT-2 capability model in about 2 hours on a single 8x H100 node, down from roughly 3 hours a month ago, driven primarily by switching the pretraining dataset from FineWeb-edu to NVIDIA ClimbMix and enabling FP8 optimizations (as reported by Karpathy). According to Karpathy, alternative datasets including Olmo, FineWeb, and DCLM produced regressions, while ClimbMix worked out of the box, suggesting immediate gains in data efficiency and reduced tuning overhead for small LLM pipelines. As reported by Karpathy, he also set up autonomous AI agents to iterate on NanoChat, making 110 changes over ~12 hours and improving validation loss from 0.862415 to 0.858039 for a d12 model without adding wall-clock time, indicating a viable pattern for continuous training-ops automation. For practitioners, this points to business opportunities in GPU cost optimization using FP8, higher-quality synthetic or curated corpora like ClimbMix for faster convergence, and agent-driven MLOps that continuously test and merge performance-improving changes.

According to Andrej Karpathy on X, nanochat now trains a GPT-2 capability model in about 2 hours on a single 8x H100 node, improved from ~3 hours a month ago, driven primarily by switching the dataset from FineWeb-edu to NVIDIA ClimbMix alongside FP8 and other tuning features (source: Andrej Karpathy on X, Mar 5, 2026). As reported by Karpathy, alternative datasets including Olmo, FineWeb, and DCLM caused regressions, while ClimbMix worked well out of the box, suggesting immediate gains in data quality and curriculum for smaller models (source: Andrej Karpathy on X). According to Karpathy, an AI agent system now continuously iterates on nanochat, making 110 changes over ~12 hours and reducing validation loss from 0.862415 to 0.858039 for a d12 model without adding wall‑clock time by running on a feature branch and merging effective ideas (source: Andrej Karpathy on X). For practitioners, the cited results highlight business opportunities in faster LLM training cycles on commodity 8x H100 nodes, data curation advantages from ClimbMix, and automation leverage via agent-driven MLOps for continuous training and deployment (source: Andrej Karpathy on X).

According to Sawyer Merritt on X, a new image post drew attention to Tesla’s AI stack and data collection, highlighting the role of on-vehicle compute and centralized training. As reported by Tesla’s 2023–2024 AI Day materials and earnings calls, Tesla is investing in Dojo to scale video model training for Full Self-Driving with billions of real-world miles as training data. According to Tesla’s 2024 Q4 update, the company continues to expand its autolabeled video datasets and multi-camera neural networks for end-to-end driving. Based on The Information’s reporting, Tesla is procuring Nvidia H100 clusters in parallel with Dojo for model training throughput. These developments create five business implications: 1) lower per-mile data acquisition costs through fleet learning; 2) faster iteration on end-to-end driving models via vertically integrated training; 3) potential licensing of autonomy stacks to OEMs once safety metrics are validated; 4) margin expansion from software subscriptions such as FSD; and 5) defensible moat from proprietary, large-scale driving video corpora. All statements are drawn from the above sources; the image post by Sawyer Merritt serves as a topical pointer to Tesla’s ongoing AI strategy.

According to Andrej Karpathy on Twitter, enabling FP8 precision training for GPT-2 using H100 GPUs has resulted in a 4.3% improvement in training time, reducing it to just 2.91 hours. Karpathy highlights that with 8xH100 spot instance pricing, the total cost to reproduce the GPT-2 model now stands at approximately $20. This marks a dramatic cost reduction compared to OpenAI's original $43,000 GPT-2 training seven years ago. As reported by Karpathy, further optimization using techniques such as Flash Attention 3 kernels, the Muon optimizer, and advanced attention patterns have contributed to these gains. While FP8 offers theoretical FLOPS advantages, Karpathy notes practical challenges including overhead from scale conversions and limited support, especially at the GPT-2 model scale. Nonetheless, the industry shift to FP8 hints at broader opportunities for cost-effective LLM training, as evidenced by torchao's reported 25% speedup on larger models like Llama3-8B. According to Karpathy, continued improvements in FP8 application and model training strategies can reduce both time and financial barriers for LLM development, opening further business and research opportunities.

According to Andrej Karpathy on Twitter, enabling FP8 training has improved 'time to GPT-2' by 4.3%, reducing the training duration to 2.91 hours on an 8x H100 GPU setup. Karpathy notes that, using spot instance pricing, the cost to reproduce GPT-2 training is now approximately $20. This marks a significant shift from GPT-2's original classification as 'too dangerous to release' in 2019 to being as accessible as MNIST today. The FP8 implementation presented practical challenges, with support limitations and real-world performance falling short of theoretical FLOPS gains. For tensorwise scaling, a speedup of about 7.3% was achieved, though Karpathy highlights that further optimizations could lower the time and cost even more. Comparatively, torchao reported a 25% speedup for Llama3-8B training using FP8. Karpathy also underscores that, thanks to advancements like Flash Attention 3 and the Muon optimizer, the cost of training GPT-2 has dropped nearly 600 times over the past seven years, offering substantial business opportunities for AI startups and researchers seeking low-cost, rapid model prototyping. As reported by Karpathy, ongoing optimizations in projects like nanochat continue to drive down training costs and times, making advanced language model training accessible to a wider audience.

According to Sawyer Merritt, Tesla announced plans to more than double the size of its onsite compute resources in Texas by the first half of 2026, measured in H100 GPU equivalents. The company aims to maximize capital efficiency by scaling its AI training infrastructure strategically, addressing training backlogs and future compute demands. This expansion signals Tesla's commitment to advancing AI-powered autonomous technologies, with significant implications for AI model training and business scalability, as reported by Sawyer Merritt on Twitter.