Research notes
Follow the questions, inspect the evidence, and revisit earlier conclusions.
From a failure to a checkable result.
Four engineering cases trace the problem, the implementation decision, and the evidence that checks the result. Read each in English or Simplified Chinese.
A browser lifecycle false negative and independent outcome scoring
A completed settings task was recorded as a failure because the browser closed before scoring. The correction preserves the page for an independent DOM check, while a separate checkout run shows why an agent's own success report cannot define the result.
Measuring two local multimodal backends with reproducible boundaries
OpenMultimodalLab preserves 612 measured attempts across 102 tasks and two pinned models. The useful engineering result is a comparison that keeps task quality, compute timing, memory, interruption recovery, and evidence provenance distinct.
Verifying a DrawDB browser history regression against frozen builds
A standalone Playwright candidate distinguished DrawDB revisions before and after Issue 687's fix. ReproLock preserved the repeated differential, build identity, and cleanup evidence, while the ordinary Playwright control exposed the limits of the product value claim.
Complete dining furniture with observed-point and native checks
The table, two backed chairs and accepted stool form four editable roots with 37 closed parts. Original-frame owner-observation gates pass; the separately authored layout certifies all 507 cross-furniture pairs at at least 5 mm separation. Full-reference table reverse-area P95 remains 48.378 mm.
First Generation,
closed with the limits visible.
Three narrow systems reached a maintained public release. This report records what each one established, what it did not establish, how to reproduce it, and why adding more features is no longer the highest-value next step.
Historical report · 2026-08-31. Adoption observations and closure decisions below describe that reporting period. Software links reflect the maintained release; the research evidence remains frozen. Reproduction examples use Windows PowerShell with a local Python 3.11–3.13 environment; each project page links the versioned setup for other platforms.
OpenMultimodalLab
Problem
Which pinned vision-language model fits one local workload and hardware budget—and can that choice be rebuilt from preserved evidence?
Why it mattered
Task and media identity, model revision, repetitions, failures, interruptions, hardware timing, and report derivation all have to remain inspectable. Otherwise a comparison becomes a screenshot rather than evidence.
What built
Versioned JSONL tasks, pinned adapters, durable attempts, deterministic task-selected scoring, strict resume, environment records, hash-bound manifests, rebuildable reports, and a maintained local Studio.
Core result
On the frozen 102-task grid and recorded RTX 4060 Laptop GPU, Qwen3-VL-2B reached mean score 0.784314 at 212.855 ms median task latency and 4,180.456 MiB peak allocated GPU memory. SmolVLM2-500M reached 0.689542, 471.531 ms, and 1,265.279 MiB. All 612 measured attempts completed without a runtime failure.
Negative result
No unfamiliar developer has produced a qualifying install, adapter, substantive Issue/PR, or independent benchmark reproduction; the four adoption-ledger counts remain 0. The frozen comparison also does not justify cross-family throughput equivalence or a universal ranking.
Established
- The exact two revisions completed the same frozen grid in the recorded environment.
- Raw attempts and manifests rebuild the committed report without another inference run.
- The offline mock path validates infrastructure and packaging.
NOT established
- A universal model ranking.
- Generalization to other workloads, hardware, or production use.
- Independent adoption or model quality from the mock path.
Reproduction
git clone --branch v1.1.2 --depth 1 https://github.com/AlbertXXuu/OpenMultimodalLab.git
cd OpenMultimodalLab
python -m venv .venv
.venv/Scripts/python.exe -m pip install -e .
.venv/Scripts/oml.exe doctor
.venv/Scripts/oml.exe run --dataset examples/tasks/smoke.jsonl --output runs/smoke-001.jsonl
.venv/Scripts/oml.exe report --input runs/smoke-001.jsonlThe mock path verifies infrastructure. Published model evidence remains bound to the immutable v1.0.0 report.
Maintenance
Correctness, security, portability, CI/dependency compatibility, reproduction clarification, and contributor onboarding remain accepted under the observed-problem gate.
Why active development stops
The bottleneck is independent use, not another local feature. More models, tasks, and interfaces would add cost before showing whether an external workflow needs them.
BrowserAgentRegression
Problem
Did a previously passing browser workflow regress after an interface or agent-facing change, and where did its outcome contract first become false?
Why it mattered
Agent self-report is not an outcome oracle. Fixtures must reset exactly, and changes to the model, harness, prompt, tools, and page must not be collapsed into one unexplained failure.
What built
Deterministic local fixtures, attempt drivers, ordered task-level DOM checkpoints, independent scoring, first-failure evidence, aggregation, and report verification. Synthetic calibration stays separate from historical real-agent smoke evidence.
Core result
The reference oracle passed all 36/36 task/variant attempts. Under the deliberately popup-blind candidate, each of the three popup conditions fell from 100% baseline success to 0%, with the first unmet checkpoint retained.
Negative result
The one historical Browser Use + DeepSeek record proves integration feasibility only. V1 does not retain enough trajectory evidence to separate observation, identification, target-selection, timing, transition, recovery, or safety causes. No unfamiliar target developer has produced a qualifying run.
Established
- The deterministic harness passes its reference cases.
- It detects and localizes the three designed synthetic regressions.
- Independent DOM checkpoints can contradict apparent completion.
NOT established
- Broad agent, model, or harness reliability.
- Causal diagnosis from first-failure outcome alone.
- Safety evaluation, product demand, or benchmark superiority.
Reproduction
git clone --branch v1.1.2 --depth 1 https://github.com/AlbertXXuu/BrowserAgentRegression.git
cd BrowserAgentRegression
python -m venv .venv
.venv/Scripts/python.exe -m pip install -e ".[dev]"
.venv/Scripts/python.exe -m playwright install chromium
.venv/Scripts/browser-agent-regression.exe doctor
.venv/Scripts/browser-agent-regression.exe demo
.venv/Scripts/browser-agent-regression.exe verify --report runs/demo-report.jsonMaintenance
Only bounded correctness, compatibility, reproduction, evidence-readability, and specifically requested named-agent integration work remains accepted.
Why active development stops
More synthetic tasks or a generic adapter framework would not answer whether a real maintainer needs the project or whether richer trajectories improve supported diagnosis.
PhysGauge
Problem
Before a video metric is used to interpret a world model, does it respond to known physical errors in a controlled system?
Why it mattered
Metric insensitivity must be separated from simulator error, initial-condition drift, visual tracking confounds, and weak-model failure. Interpretation has to follow the registered capability gates.
What built
An analytically verified equal-mass two-disc world, six controlled candidate families, state-grounded checks, four visual metrics, frozen severity sweeps, hash-bound evidence, and a separate three-seed learned-predictor study with a preregistered stop rule.
Core result
All 24 oracle rollouts passed. State-grounded metrics detected every injected violation, and all four included visual metrics increased monotonically on the frozen inelastic and momentum-kick sweeps across 240 records. Frame-order reversal remained a bounded counterexample: Pixel Fréchet stayed between 0 and 5.76e-15 while the kinematic residual was large.
Negative result
R2's three predictors produced 91.8%–99.6% post-contact partial-error rates. All were classified too-weak; the preregistered outcome is inconclusive-model, and R3 did not start. Stored visual disagreement cannot support a learned-model blind-spot claim because capability failed first.
Established
- The registered analytic world and evidence chain behave as recorded.
- The included metrics have the bounded v1 sensitivity behavior above.
- The R2 pipeline correctly stops at its capability gate.
NOT established
- Reliability on a sufficiently capable learned world model.
- A learned-model visual blind spot from R2.
- Generalization to OOD physics, pixels-only systems, or real video.
Reproduction
git clone --branch v1.1.2 --depth 1 https://github.com/AlbertXXuu/PhysGauge.git
cd PhysGauge
python -m venv .venv
.venv/Scripts/python.exe -m pip install -e .
.venv/Scripts/physgauge.exe doctor
.venv/Scripts/physgauge.exe run --output runs/my-calibration
.venv/Scripts/physgauge.exe verify --bundle docs/evidence/v1.0.0Maintenance
Correctness, compatibility, and reproduction fixes remain allowed. New learned-model validation requires a new protocol, unseen test split, predeclared capability gate, and bounded cost.
Why active development stops
R2 already answered whether its planned comparison was valid: the model was too weak. Tuning on that test evidence would be post-hoc; a legitimate continuation must start from a new need and unseen data.
What AlvenX learned
Identity comes before numbers.
Tasks, inputs, revisions, environments, repetitions, failures, and derivations must be bound before a metric becomes inspectable evidence.
The oracle must sit outside the claim.
Agent self-report cannot prove application state. A visual distance cannot prove physical correctness. Independent contracts are narrow by design.
Validate response before interpretation.
Known changes and capability gates come first. If a gate fails, the harder conclusion stays unanswered.
A justified stop is a result.
Retained failures, unknown causes, zero adoption evidence, and inconclusive experiments constrain the next decision instead of being polished away.
Closure decision
Software evolves at v1.1.2; research history remains immutable at v1.0.0. The remaining limit is external evidence: all three projects have one owner and no qualifying stranger use. Until external use or a newly registered technical question reveals a concrete problem, First Generation stays in maintenance.