Google Launches Multimodal Knowledge Agent Benchmark for Visual Tasks

Google, in collaboration with Artificial Analysis, has launched the Multimodal Knowledge Agent (MKA) benchmark to evaluate how AI agents handle visually complex tasks in professional workflows. The benchmark aims to address the limitations of text-only benchmarks by focusing on tasks that require reasoning across multimodal inputs, such as images, page layouts, and spreadsheets.

Benchmark Design and Scope

The MKA benchmark was developed with input from domain experts across 10 industries and 83 occupations, including architects, biologists, underwriters, and data scientists. The evaluation set consists of 200 tasks that simulate real-world workflows, each requiring three to eight hours of effort from a seasoned professional. Tasks are paired with multimodal input files, realistic prompts, and an expert-written rubric comprising approximately 70 yes/no checks. The rubric weights factual accuracy at 75% and visual design at 25%.

To ensure fairness and consistency, all models are evaluated using the open-source agent harness Stirrup, which provides a standardized set of tools, a sandbox environment, and a fixed turn budget. Deliverables are graded by an agentic autorater built on a non-Google model, which achieved a 96% agreement rate with human domain experts during testing. To prevent data leakage, the 200-task evaluation set is kept private, while a separate set of 10 public tasks is available on Hugging Face for broader experimentation.

Performance Insights and Gemini 4 Carbon

Google's technical report highlights significant variability in model performance on the MKA benchmark. Frontier models demonstrated differences in efficiency, requiring between 30 and 130 turns on average to complete tasks. The report underscores the importance of strong perception capabilities for success in visually complex workflows.

Among the tested models, Gemini 4 Carbon emerged as a standout performer, ranking second in quality while achieving six times lower costs compared to the top-scoring model. Positioned on the Pareto frontier of quality and price, Gemini 4 Carbon is now accessible via the Gemini API in Google AI Studio, with updated developer documentation available for integration.

Next Steps

Researchers and developers can explore the MKA benchmark and evaluate their models on Artificial Analysis. The initiative marks a significant step toward advancing AI's ability to handle real-world, multimodal professional tasks, with implications for industries reliant on visually intensive workflows.