Open
A local-first evaluation pipeline for choosing a pinned vision-language model from inspectable evidence.
- 102
- reviewed tasks
- 612
- measured attempts
- 0
- runtime failures
Problem
Model demos and aggregate leaderboards do not show which pinned vision-language model fits a specific local workload and hardware budget—or whether the conclusion can be rebuilt from preserved evidence.
Why it matters
A useful comparison must hold task identity, media, model revision, repetitions, failures, and hardware timing still enough to inspect. It must also rebuild its report without paying for another inference run.
Core contribution
OpenMultimodalLab binds versioned tasks, interchangeable adapters, durable raw attempts, deterministic task-selected scoring, and environment-aware reports into one reproducible workflow.
- Raw JSONL and hash-bound manifests remain authoritative; summaries are derived.
- Task-selected deterministic scoring keeps the frozen corpus independent of an LLM judge.
- Strict resume verifies the exact durable prefix before continuing an interrupted run.
Hero result
On the recorded RTX 4060 Laptop GPU, Qwen3-VL-2B reached a 0.784 mean task score at 212.9 ms median task latency; SmolVLM2-500M reached 0.690 at 471.5 ms while using about 70% less peak allocated GPU memory.
How it works
- 01
Version inputs
Validate task JSONL, preserve media identity, and pin model revisions.
- 02
Run completely
Separate warm-up from measured repetitions and retain every success, failure, and retry.
- 03
Score deterministically
Apply the scorer selected by each task instead of asking another model to judge it.
- 04
Rebuild evidence
Verify hashes and formal-grid completeness, then derive the Markdown, CSV, failure, and SVG report bundle.
Evidence
Both pinned models completed 306 measured attempts over the same 102-task grid with zero runtime failures. The committed report records score, latency, memory, model revision, task/media hashes, and the bounded interpretation.
Limitations
- 01
The comparison covers two pinned revisions, one recorded hardware profile, and a reviewed synthetic workload—not a universal model ranking.
- 02
Cross-family token throughput is not treated as directly equivalent.
- 03
Independent adoption and generalization beyond the frozen task grid are not established.
Reproduce
The current v1.1.2 software includes a dependency-free mock path. It verifies the evaluation infrastructure, not model quality; the published result remains bound to v1.0.0.
Windows PowerShell · Python 3.11–3.13. Run from a new checkout; each command uses its local virtual environment.
git clone --branch v1.1.2 --depth 1 https://github.com/AlbertXXuu/OpenMultimodalLab.git
cd OpenMultimodalLab
python -m venv .venv
.venv/Scripts/python.exe -m pip install -e .
.venv/Scripts/oml.exe doctor
.venv/Scripts/oml.exe run --dataset examples/tasks/smoke.jsonl --output runs/smoke-001.jsonl
.venv/Scripts/oml.exe report --input runs/smoke-001.jsonlRelease & maintenance
Closure maintenance patch.
Immutable protocol, evidence, and claims.
The v1.1.2 maintenance patch aligns the Studio navigation with the locked AlvenX product-header shell and clarifies measurement boundaries without rerunning the study. The v1.0.0 evidence remains unchanged. Future tasks, adapters, schemas, or UI surfaces require a reproducible issue, experiment failure, user need, or registered hypothesis.
Read the maintenance policy