List of AI News about GSM8k
| Time | Details |
|---|---|
|
2026-11-04 13:26 UTC |
Stanford shows hinted scaling forecasts GSM8K accuracy
According to Stanford AI Lab via X, researchers led by Suze van Adrichem report that hinted scaling laws can forecast GSM8K accuracy within 3 points using 5600x less compute by jointly modeling compute scale and hint level. |
|
2026-11-03 16:04 UTC |
Stanford AI Lab unveils hinted scaling laws for GSM8K forecasts
According to Stanford AI Lab, researchers led by Suze van Adrichem show that hinted scaling laws across hint level H and compute scale C forecast GSM8K accuracy within 3 points using 5600x less compute, as reported on X. |
|
2026-11-03 11:49 UTC |
Stanford AI Lab unveils hinted scaling laws to forecast GSM8K
According to Stanford AI Lab citing work by Suze van Adrichem and co-leads Luke Bailey and Raghav Pudi, hinted scaling laws forecast GSM8K accuracy within 3 points using 5600x less compute by modeling compute scale and hint level. |
|
2026-03-27 02:57 UTC |
MEMCOLLAB Breakthrough: Cross-Model Memory Boosts Llama 3 8B to 42.4% on MATH500 — Analysis and Business Impact
According to God of Prompt, Pennsylvania State University identified that agent memories distilled from a single model’s reasoning traces carry model-specific biases and heuristics that hurt transfer, causing performance to fall below zero-memory baselines when moved across models; as reported by the tweet and summarized from the study highlights, giving a 7B model’s memory to a 32B model reduced MATH500 from 63.8% to 50.6% and HumanEval from 68.3% to 34.1%, and the reverse transfer also degraded results. According to the same source, the proposed fix, MEMCOLLAB, constructs memory from cross-model agreement by contrasting a success trajectory with a failure trajectory to extract invariant reasoning principles, not style; this raised Llama 3 8B MATH500 from 27.4% to 42.4% and lifted average accuracy across four benchmarks from 41.7% to 53.9%. As reported by God of Prompt, Qwen 7B improved from 52.2% to 67.0% on MATH500 and from 42.7% to 74.4% on HumanEval, while reasoning turns dropped from 3.3 to 1.5 on HumanEval and 3.1 to 1.4 on MBPP, indicating efficiency gains that reduce inference cost. According to the same source, cross-architecture memory construction (Qwen 32B plus Llama 8B) outperformed same-family memory on GSM8K at 95.2% vs 93.6%, signaling opportunities for vendors to standardize cross-model memory pipelines, lower token spend, and improve reliability in production agents for coding, math tutoring, and workflow automation. |
|
2026-02-04 01:35 UTC |
Latest Analysis: Phi and Mistral Models Show 13% Accuracy Drop on GSM1k vs GSM8k, Revealing Memorization Issues
According to God of Prompt on Twitter, recent testing shows that the Phi and Mistral models experienced a significant 13% accuracy drop when evaluated on the GSM1k benchmark compared to GSM8k. Some model variants saw drops as high as 13.4 percentage points. The analysis suggests these models are not demonstrating true reasoning abilities but rather memorization, as they were exposed to the correct answers during training. This finding highlights critical concerns about the generalization and reliability of these AI models for business and research applications. |
|
2026-02-04 01:35 UTC |
Latest Analysis Reveals 0.32 Correlation Between GSM8k Reproduction and Performance Gap in AI Models
According to God of Prompt on Twitter, researchers have identified a 0.32 correlation between an AI model's ability to reproduce GSM8k test examples and its performance gap. This finding suggests that models which can recite test questions tend to perform worse when faced with new, unseen questions. As reported by God of Prompt, the implication is that these models may be memorizing answers rather than demonstrating true problem-solving capabilities, raising concerns about the validity of current AI evaluation benchmarks. |