List of AI News about evaluation
| Time | Details |
|---|---|
|
2026-11-17 16:55 UTC |
Critics challenge Pangram’s non-native English datasets
According to a post highlighted by Timnit Gebru on X, ML Kejera criticized Pangram for claiming robustness based on four non-native English datasets, arguing this approach flattens linguistic diversity and undermines credible evaluation. |
|
2026-11-04 09:00 UTC |
DeepLearning.AI launches agent evaluation course with Arize
According to DeepLearning.AI, a new course called Evaluating and Improving Agentic Applications, built with Arize AI and taught by Laurie Voss, teaches practitioners to assess agent behavior beyond single successful runs. |
|
2026-09-26 07:30 UTC |
Andrew Ng Outlines stage-based AI engineering
According to DeepLearningAI, Andrew Ng urges stage-based AI tactics for reliability, covering eval pipelines, scalable architecture, and feedback loops. |
|
2026-09-24 09:49 UTC |
METR Chart Replacement Sparks 2026 Analysis
According to @emollick, a new visual should replace METR’s Long Task Horizon chart, prompting fresh analysis of AI capability benchmarks. |
|
2026-09-16 06:40 UTC |
AI benchmarks Crisis Exposed by New Paper
According to @emollick, public AI benchmarks are saturated or error ridden, severely underestimating current model abilities, as the cited paper shows. |
|
2026-09-08 06:04 UTC |
METR Long-Horizon Benchmarks Face Saturation
According to emollick, METR’s long-horizon metric may be saturated as studies report 18+ weeks of agent work from Fable in harnesses. |
|
2026-09-06 07:17 UTC |
DeepLearningAI Reveals 5 Coding-Agent Skills
According to DeepLearningAI, top teams curb agent autonomy with human review, iterative plans, and verification to cut costs and failures. |
|
2026-08-16 09:33 UTC |
Human Judgments Drive AI Evaluation Breakthrough
According to emollick, human opinions are essential for benchmarking non-verifiable AI tasks, urging adoption of qualitative research methods. |
|
2026-08-01 10:19 UTC |
LLMs Improve Across Unverifiable Tasks
According to @emollick, LLMs advance in formal domains also lift performance in less-verifiable fields, easing concerns over non-checkable answers. |
|
2026-07-29 07:45 UTC |
AI Behavioral Observatory Enables Valid Prompt Testing
According to @emollick, the AI Behavioral Observatory open source tool runs statistically valid tests on model behavior across prompts. |
|
2026-07-18 15:24 UTC |
Kimi K3 Reveals English Chain-of-Thought Bias
According to emollick, Kimi K3 used 95.5% English in its chain-of-thought for a Chinese prompt, signaling training bias and evaluation gaps. |
|
2026-07-08 20:13 UTC |
OpenAI Benchmarks Shake-up Spurs 2026 Analysis
According to emollick, OpenAI questioned coding evals yet has not shared GPT5.6 GDPval, raising transparency and capability tracking concerns. |
|
2026-07-08 12:45 UTC |
OpenAI Audits SWE-Bench Pro, flags 70% noise
According to @OpenAI, SWE-Bench Pro shows a ~70% noise ceiling and no longer reliably measures frontier coding capability, urging a shift to new evals. |
|
2026-07-06 17:41 UTC |
Ethan Mollick Says prompting tricks fade, management wins
According to @emollick, prompt hacks add little value; define goals, outputs, quality bars, and tests to get ROI from AI systems. |
|
2026-06-16 09:23 UTC |
OpenAI Evals Reform Guides Next Benchmarks
According to OpenAI on X, leaders discuss better evals to forecast model progress as saturated benchmarks get gamed, outlining next judgment areas. |
|
2026-06-03 08:38 UTC |
Leni AI Elevates Verified Investment Analysis
According to @godofprompt, Leni targets finance with verified outputs and rigorous reasoning to cut costly errors in institutional investment workflows. |
|
2026-05-16 04:57 UTC |
iFixAI Launches 32-Test Safety Score for AI
According to @godofprompt, iFixAI runs 32 tests on deployed AI to score hallucinations, manipulation, and consistency, offering actionable evals for teams. |
|
2026-04-30 08:14 UTC |
GPT5.5 Tops Benchmarks yet Misfires Often
According to @godofprompt, AA-Omniscience shows GPT-5.5 ranks highest for smarts but is most confidently wrong when penalized for guessing. |
|
2026-04-28 05:25 UTC |
Test of Time LLM Debuts With Retro Benchmark Fun
According to @soumithchintala, Test of Time LLM offers a playful, retro-style benchmark link, highlighting community interest in evaluators. |
|
2026-04-21 11:12 UTC |
LLM Judge Bias Exposed: New Position Bias Benchmark Shows Up To 66% Flip Rate — 2026 Analysis
According to Ethan Mollick on X (Twitter), large language models used as judges display significant position bias, with judgments flipping when answer order is swapped; he cites Lech Mazur’s New LLM Position Bias Benchmark showing a median 45% flip rate on decisive pairs and a reported 66% flip rate for GPT-5.4 (as reported by Lech Mazur’s thread and benchmark summary). According to Mollick, simple presentation changes materially alter outcomes, indicating current LLM-as-judge pipelines remain unreliable without controls (as reported by Ethan Mollick). According to Lech Mazur, mitigation via better harnessing—multiple judging runs, randomized order, and aggregation—can reduce variance, suggesting practical steps for enterprise evaluation workflows and AI product A/B testing. Business impact: according to Mollick’s post, organizations relying on LLM judges for qualitative assessments (creative scoring, code review, search ranking, and RLHF data curation) should add randomized comparisons, majority voting, and calibration audits to improve consistency and reduce bias-induced risk. |