Browser
A local-first harness that detects browser workflow regressions at the first failed outcome contract.
- 3
- controlled tasks
- 4
- page variants
- 36/36
- reference passes
Problem
A browser-agent stack can change while its happy-path demo still succeeds. A final pass/fail label does not reveal whether a previously passing behavior regressed or where the workflow first departed from its contract.
Why it matters
Agent self-report is not an outcome oracle. Browser fixtures must reset exactly, while model, harness, prompt, tool, and UI changes must remain distinguishable rather than being collapsed into one apparent failure.
Core contribution
BrowserAgentRegression separates resettable local fixtures, attempt drivers, independent DOM checkpoints, first-failure evidence, and aggregation so the result does not depend on the agent claiming success.
- Freeze task-level outcome contracts rather than brittle selectors or agent prose.
- Calibrate the harness with deterministic reference and deliberately degraded drivers.
- Recompute summaries during verification and keep synthetic calibration separate from real-agent evidence.
Hero result
The reference oracle passed all 36 task/variant attempts. Under the deliberately popup-blind candidate, the three popup conditions fell from 100% baseline success to 0%, with consistent first-failure localization.
How it works
- 01
Reset a fixture
Serve one controlled task and page variant from a deterministic local browser environment.
- 02
Run attempts
Apply the reference or candidate driver repeatedly while preserving exact task and configuration identity.
- 03
Score independently
Evaluate ordered DOM outcome checkpoints instead of trusting the driver's final statement.
- 04
Localize divergence
Compare baseline and candidate summaries and retain the first unmet checkpoint with expected and observed state.
Evidence
The frozen v1 calibration establishes that the harness detects its designed synthetic popup-blind regression and localizes the first failed checkpoint. It validates the calibration design, not general browser-agent quality.
Limitations
- 01
Three synthetic tasks, four controlled variants, and deterministic drivers do not establish broad model reliability.
- 02
The historical one-run Browser Use + DeepSeek record proves integration feasibility only.
- 03
Product demand and superiority over BrowserGym, WebArena, or a team's own eval stack are not established.
Reproduce
The current v1.1.2 software runs a deterministic 12-attempt demo locally and verifies the generated report with the same validator used in CI; the published calibration remains bound to v1.0.0.
Windows PowerShell · Python 3.11–3.13. Run from a new checkout; each command uses its local virtual environment.
git clone --branch v1.1.2 --depth 1 https://github.com/AlbertXXuu/BrowserAgentRegression.git
cd BrowserAgentRegression
python -m venv .venv
.venv/Scripts/python.exe -m pip install -e ".[dev]"
.venv/Scripts/python.exe -m playwright install chromium
.venv/Scripts/browser-agent-regression.exe doctor
.venv/Scripts/browser-agent-regression.exe demo
.venv/Scripts/browser-agent-regression.exe verify --report runs/demo-report.jsonRelease & maintenance
Closure maintenance patch.
Immutable protocol, evidence, and claims.
The v1.1.2 maintenance patch makes the checkpoint evidence rows equal-height and keeps the first-divergence label readable at every supported Studio width. The v1.0.0 protocol and evidence remain unchanged. New tasks, mutations, generic adapters, schemas, or UI surfaces must pass the project's observed-problem gate.
Read the maintenance policy