Skip to main content
AI News List

List of AI News about evaluation

Time Details
2026-11-17
16:55 UTC
Critics challenge Pangram’s non-native English datasets

According to a post highlighted by Timnit Gebru on X, ML Kejera criticized Pangram for claiming robustness based on four non-native English datasets, arguing this approach flattens linguistic diversity and undermines credible evaluation.

Source
2026-11-04
09:00 UTC
DeepLearning.AI launches agent evaluation course with Arize

According to DeepLearning.AI, a new course called Evaluating and Improving Agentic Applications, built with Arize AI and taught by Laurie Voss, teaches practitioners to assess agent behavior beyond single successful runs.

Source
2026-09-26
07:30 UTC
Andrew Ng Outlines stage-based AI engineering

According to DeepLearningAI, Andrew Ng urges stage-based AI tactics for reliability, covering eval pipelines, scalable architecture, and feedback loops.

Source
2026-09-24
09:49 UTC
METR Chart Replacement Sparks 2026 Analysis

According to @emollick, a new visual should replace METR’s Long Task Horizon chart, prompting fresh analysis of AI capability benchmarks.

Source
2026-09-16
06:40 UTC
AI benchmarks Crisis Exposed by New Paper

According to @emollick, public AI benchmarks are saturated or error ridden, severely underestimating current model abilities, as the cited paper shows.

Source
2026-09-08
06:04 UTC
METR Long-Horizon Benchmarks Face Saturation

According to emollick, METR’s long-horizon metric may be saturated as studies report 18+ weeks of agent work from Fable in harnesses.

Source
2026-09-06
07:17 UTC
DeepLearningAI Reveals 5 Coding-Agent Skills

According to DeepLearningAI, top teams curb agent autonomy with human review, iterative plans, and verification to cut costs and failures.

Source
2026-08-16
09:33 UTC
Human Judgments Drive AI Evaluation Breakthrough

According to emollick, human opinions are essential for benchmarking non-verifiable AI tasks, urging adoption of qualitative research methods.

Source
2026-08-01
10:19 UTC
LLMs Improve Across Unverifiable Tasks

According to @emollick, LLMs advance in formal domains also lift performance in less-verifiable fields, easing concerns over non-checkable answers.

Source
2026-07-29
07:45 UTC
AI Behavioral Observatory Enables Valid Prompt Testing

According to @emollick, the AI Behavioral Observatory open source tool runs statistically valid tests on model behavior across prompts.

Source
2026-07-18
15:24 UTC
Kimi K3 Reveals English Chain-of-Thought Bias

According to emollick, Kimi K3 used 95.5% English in its chain-of-thought for a Chinese prompt, signaling training bias and evaluation gaps.

Source
2026-07-08
20:13 UTC
OpenAI Benchmarks Shake-up Spurs 2026 Analysis

According to emollick, OpenAI questioned coding evals yet has not shared GPT5.6 GDPval, raising transparency and capability tracking concerns.

Source
2026-07-08
12:45 UTC
OpenAI Audits SWE-Bench Pro, flags 70% noise

According to @OpenAI, SWE-Bench Pro shows a ~70% noise ceiling and no longer reliably measures frontier coding capability, urging a shift to new evals.

Source
2026-07-06
17:41 UTC
Ethan Mollick Says prompting tricks fade, management wins

According to @emollick, prompt hacks add little value; define goals, outputs, quality bars, and tests to get ROI from AI systems.

Source
2026-06-16
09:23 UTC
OpenAI Evals Reform Guides Next Benchmarks

According to OpenAI on X, leaders discuss better evals to forecast model progress as saturated benchmarks get gamed, outlining next judgment areas.

Source
2026-06-03
08:38 UTC
Leni AI Elevates Verified Investment Analysis

According to @godofprompt, Leni targets finance with verified outputs and rigorous reasoning to cut costly errors in institutional investment workflows.

Source
2026-05-16
04:57 UTC
iFixAI Launches 32-Test Safety Score for AI

According to @godofprompt, iFixAI runs 32 tests on deployed AI to score hallucinations, manipulation, and consistency, offering actionable evals for teams.

Source
2026-04-30
08:14 UTC
GPT5.5 Tops Benchmarks yet Misfires Often

According to @godofprompt, AA-Omniscience shows GPT-5.5 ranks highest for smarts but is most confidently wrong when penalized for guessing.

Source
2026-04-28
05:25 UTC
Test of Time LLM Debuts With Retro Benchmark Fun

According to @soumithchintala, Test of Time LLM offers a playful, retro-style benchmark link, highlighting community interest in evaluators.

Source
2026-04-21
11:12 UTC
LLM Judge Bias Exposed: New Position Bias Benchmark Shows Up To 66% Flip Rate — 2026 Analysis

According to Ethan Mollick on X (Twitter), large language models used as judges display significant position bias, with judgments flipping when answer order is swapped; he cites Lech Mazur’s New LLM Position Bias Benchmark showing a median 45% flip rate on decisive pairs and a reported 66% flip rate for GPT-5.4 (as reported by Lech Mazur’s thread and benchmark summary). According to Mollick, simple presentation changes materially alter outcomes, indicating current LLM-as-judge pipelines remain unreliable without controls (as reported by Ethan Mollick). According to Lech Mazur, mitigation via better harnessing—multiple judging runs, randomized order, and aggregation—can reduce variance, suggesting practical steps for enterprise evaluation workflows and AI product A/B testing. Business impact: according to Mollick’s post, organizations relying on LLM judges for qualitative assessments (creative scoring, code review, search ranking, and RLHF data curation) should add randomized comparisons, majority voting, and calibration audits to improve consistency and reduce bias-induced risk.

Source