According to DeepLearningAI, Andrew Ng urges stage-based AI tactics for reliability, covering eval pipelines, scalable architecture, and feedback loops.

According to @emollick, a new visual should replace METR’s Long Task Horizon chart, prompting fresh analysis of AI capability benchmarks.

According to @emollick, public AI benchmarks are saturated or error ridden, severely underestimating current model abilities, as the cited paper shows.

According to emollick, METR’s long-horizon metric may be saturated as studies report 18+ weeks of agent work from Fable in harnesses.

According to DeepLearningAI, top teams curb agent autonomy with human review, iterative plans, and verification to cut costs and failures.

According to emollick, human opinions are essential for benchmarking non-verifiable AI tasks, urging adoption of qualitative research methods.

According to @emollick, LLMs advance in formal domains also lift performance in less-verifiable fields, easing concerns over non-checkable answers.

According to @emollick, the AI Behavioral Observatory open source tool runs statistically valid tests on model behavior across prompts.

According to emollick, Kimi K3 used 95.5% English in its chain-of-thought for a Chinese prompt, signaling training bias and evaluation gaps.

According to emollick, OpenAI questioned coding evals yet has not shared GPT5.6 GDPval, raising transparency and capability tracking concerns.

According to @OpenAI, SWE-Bench Pro shows a ~70% noise ceiling and no longer reliably measures frontier coding capability, urging a shift to new evals.

According to @emollick, prompt hacks add little value; define goals, outputs, quality bars, and tests to get ROI from AI systems.

According to OpenAI on X, leaders discuss better evals to forecast model progress as saturated benchmarks get gamed, outlining next judgment areas.

According to @godofprompt, Leni targets finance with verified outputs and rigorous reasoning to cut costly errors in institutional investment workflows.

According to @godofprompt, iFixAI runs 32 tests on deployed AI to score hallucinations, manipulation, and consistency, offering actionable evals for teams.

According to @godofprompt, AA-Omniscience shows GPT-5.5 ranks highest for smarts but is most confidently wrong when penalized for guessing.

According to @soumithchintala, Test of Time LLM offers a playful, retro-style benchmark link, highlighting community interest in evaluators.

According to Ethan Mollick on X (Twitter), large language models used as judges display significant position bias, with judgments flipping when answer order is swapped; he cites Lech Mazur’s New LLM Position Bias Benchmark showing a median 45% flip rate on decisive pairs and a reported 66% flip rate for GPT-5.4 (as reported by Lech Mazur’s thread and benchmark summary). According to Mollick, simple presentation changes materially alter outcomes, indicating current LLM-as-judge pipelines remain unreliable without controls (as reported by Ethan Mollick). According to Lech Mazur, mitigation via better harnessing—multiple judging runs, randomized order, and aggregation—can reduce variance, suggesting practical steps for enterprise evaluation workflows and AI product A/B testing. Business impact: according to Mollick’s post, organizations relying on LLM judges for qualitative assessments (creative scoring, code review, search ranking, and RLHF data curation) should add randomized comparisons, majority voting, and calibration audits to improve consistency and reduce bias-induced risk.

According to @emollick referencing The New York Times by Mike Isaac, Google’s AI Overviews show roughly 90 percent accuracy but a consequential 10 percent error rate at Google’s multi‑trillion annual search scale, highlighting why evaluating AI quality is hard when identical errors also exist in sources like Wikipedia and source traceability is weaker in AI answers. As reported by The New York Times, the case study shows that AI Overviews can surface useful synthesized answers that many users might not find on their own, yet inconsistent citation visibility complicates verification and accountability. According to The New York Times, this creates operational risk for publishers, brands, and advertisers that rely on search accuracy, while opening opportunities for enterprise evaluation tooling, retrieval‑augmented generation pipelines with explicit citation, and content provenance standards to improve auditability.

According to Yann LeCun on X, resharing Steven Pinker’s video on media negativity bias highlights how selective bad-news framing skews public risk perception; for AI builders, this underscores the need for calibrated communication and evidence-based benchmarks in AI safety, deployment metrics, and policy debates (as reported by the linked YouTube video from Steven Pinker). According to Steven Pinker’s YouTube presentation, negative selection and availability bias make people overestimate systemic collapse, a dynamic that can also distort narratives around AI risk, automation impact, and model failures; AI teams can counter this by publishing longitudinal reliability data, post-deployment incident rates, and audited evaluation suites. As reported by the original X post from Yann LeCun, reframing with trend data can improve stakeholder trust; AI companies can apply this by standardizing model cards, red-teaming disclosures, and quarterly safety and performance reports tied to concrete baselines.