Measuring two local multimodal backends with reproducible boundaries

A score alone cannot tell a reader which inputs, decoding settings, or hardware produced it. Timing also changes meaning when preprocessing, first token computation, and full task completion are mixed. OpenMultimodalLab addresses that problem with versioned tasks, explicit metric definitions, and preserved records rather than a standalone score chart.

The formal v1.0.0 evidence uses 102 unique human checked synthetic tasks across image, document, short video, and robustness datasets. Qwen3-VL-2B-Instruct and SmolVLM2-500M-Video-Instruct each complete three measured repetitions: 306 attempts per backend and 612 in total. One successful warm up per backend is stored separately and excluded from measured aggregates.

Both backends ran on one NVIDIA RTX 4060 Laptop GPU with 8,188 MiB, batch size one, greedy decoding, no retries, and no reloads during measurement. Their model revisions are immutable: Qwen at 89644892e4d85e24eaac8bacfd4f463576704203 and SmolVLM2 at 7b375e1b73b11138ff12fe22c8f2822d8fe03467.

A successful attempt means inference and evaluation completed. It does not mean the answer was fully correct. The report therefore publishes runtime failures and deterministic task scores separately. On this grid, both backends had zero runtime failures; mean task scores were 0.784 and 0.690. Category results remain available because an aggregate can conceal different strengths.

The Transformers adapter synchronizes CUDA before generation and when the first generated token logits are available. TTFT covers that local compute interval: its medians are 120.5 ms for Qwen and 260.0 ms for SmolVLM2. Task latency measures the adapter invocation plus deterministic evaluation, so it includes work outside generation and has a different boundary.

Memory is torch.cuda.max_memory_allocated during generation after resetting peak statistics. It measures PyTorch allocated CUDA memory, not total process VRAM or the entire device peak. Throughput counts each model's native token IDs; different tokenizers make a cross family tokens per second ranking less informative than quality, latency, and failures.

Each JSONL record is flushed and synced before an atomic manifest checkpoint records counts, bytes, and SHA-256. Strict resume checks dataset and media hashes, task order, model revision, generation settings, environment, and the exact prefix of the phase and repetition plan. A changed configuration, truncated final record, or mismatch with the manifest is rejected; recovery never guesses which rows to keep.

The report builder validates complete grids, clean source commits, model identity, input hashes, and JSONL sidecars before aggregation. Its bundle binds output and generator hashes and can be rebuilt from preserved results without another model call. The recorded result filenames carry August 10; their manifests retain the actual UTC execution timestamps separately. Article publication on October 2 does not imply a new benchmark run.

These measurements apply to the preserved synthetic corpus, pinned models, recorded machine, and decoding settings. Three repetitions improve visibility into this experiment but do not establish a general model ranking or production quality. SHA-256 detects inconsistency while the manifest is trusted; it is not a signature against an actor who can rewrite both artifacts.

LMMs-Eval is a primary reference for reusable multimodal task and model evaluation. The narrower contribution here is making a small local comparison inspectable through declared boundaries, strict recovery, and report reconstruction. Its numbers are supported by the preserved OpenMultimodalLab records.