BreakingWeb

Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

BreakingWeb evaluates browser agents on matched clean/intervention tasks across seven self-hosted web environments. The benchmark contains 519 task pairs and 29 intervention families. Each pair preserves the user instruction, latent target, and backend success criterion.

Paper · Website · Code · Live demo

Evaluation artifacts

• breakingweb-results-v2: Browser-Use trajectories for six models.
• breakingweb-results-v3: additional Browser-Use runs and screenshot-only BrowserGym trajectories.

Each dataset card describes its contents and how to download the artifacts.

Citation

@misc{yin2026constructingchallengingbrowserusetasks,
      title={Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions},
      author={Xunjian Yin and Tianchen Guan and Jinao Wang and Weili Cao and Daisy Xinlei Lin and Royce Cheng-Yue and Keagan Long and Kyle Wong and Bhuwan Dhingra and Xiangjun Wang and Shuyan Zhou},
      year={2026},
      eprint={2609.35814},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.35814},
}