Research
Three results that build on each other
The through line is a single question posed three ways: first measuring where humans
look, then building a model that reproduces it from scratch, then asking how far
today's vision-language models still are from human visual competence.
01Measurement
Nature Communications · 2026
Eye movements during free viewing to maximize scene understanding
- Psychophysics · Experimental design
- EyeLink 1000 Plus eye tracking
- Fixation/saccade parsing · Time-series analysis
- Qualtrics
- GLMs · Statistical modeling
- Python
What humans look at when freely viewing a scene is not well understood. We measured
eye movements under different instructions while observers viewed Winograd images:
pairs differing by a small visual alteration that greatly changes how the scene is
interpreted. Free-viewing fixations resemble those of observers describing scenes,
and differ from those counting or searching for objects. They are directed toward the
people and objects whose removal most alters scene interpretation, rather than toward
the most salient or most meaningfully judged regions. Constraining observers to fixate
objects irrelevant to understanding degrades their descriptions, showing that
free-viewing eye movements are functionally important for scene comprehension.
Applied value. A principled account of where people
look under free viewing gives interface, ad, and content designers a reason for their
layout decisions, and gives eye-tracking-based UX research a benchmark that isn't
pure saliency.
02Model
Under review · Nature Human Behaviour
Why we look where we look: emergent human-like fixations of a foveated visual language model
- Reinforcement learning · REINFORCE
- Policy-gradient optimization
- Vision-language models (DeepSeek Janus, OVIS2.5)
- Embedding extraction · vLLM · HuggingFace
- PyTorch
- GPU computing (A6000) · Distributed training
When humans view scenes without a specific task, they first direct their eye
movements toward the scene center, then fixate on people, text, objects being looked
at or grasped, and semantically meaningful regions. What these signature fixation
patterns reflect, and whether they optimize an underlying perceptual task, has
remained unknown. We show that a computational agent with simulated foveation,
trained to optimize scene comprehension, exhibits emergent human fixation signature
patterns. Versions of the agent trained instead to search or classify scenes, or
equipped with peripheral vision better or worse than human vision, predicted human
fixation patterns less accurately. Human free-viewing fixation patterns may therefore
emerge as a functional byproduct of optimizing scene comprehension under the
biological constraints of foveated vision.
Applied value. A model that decides where to look
next processes only a fraction of the pixels a full-resolution model does, which
matters for on-device VLMs, robotics, AR/VR, and any perception system with a
compute or bandwidth budget.
03Benchmark
arXiv · 2026
Evolution of accuracy and visual-cognitive errors in a decade of vision-language AI models
- Vision-language models (Gemini, GPT-4 series)
- Computer vision preprocessing
- Similarity metrics
- Behavioral ML evaluation
- Error analysis
- Large-scale inference · vLLM · HuggingFace
Visual reasoning benchmarks mostly use simple scenes, few human descriptions, and
rarely ask what models actually get wrong. We introduce the Complex Social Behavior
dataset, 100 images depicting complex social interactions, and trace scene
description accuracy across a decade of vision-language models against 20 human
descriptions and a gold standard. Multimodal LLMs now reach accuracies similar to the
top-ranked human descriptions, and have closed the gap between simple MS-COCO scenes
and scenes depicting complex behaviors. Across five visual-cognitive error types,
nearly all have been eliminated, except that models still occasionally rely on
different image regions than humans do.
Applied value. An error taxonomy for VLMs on complex
social scenes tells product teams which failure modes have been solved and which
still surface in deployment, which is more actionable than a single accuracy number.
