AssemblyAI's voice-standard-bench is a Go command-line tool that benchmarks every model behind the llm_gateway across four distinct tasks: summarization, information extraction, text correction, and dictation. The framework runs these models against real AssemblyAI call transcripts, or a synthetic dictation dataset for the dictation task, tracking both latency and quality metrics broken down by audio length.

Users set their API token via the ASSEMBLY_AI_TOKEN environment variable; there is no -api-key flag, so the credential never appears in process lists or shell history. Commands are run from the src/ directory, where normalized/, results/, results_ttft/, errors/, and errors_ttft/ directories reside one level up at the repository root. The tool is invoked with go run . and supports a -h flag that lists every available flag with a one-line description.

For each combination of model, task, and transcript, the framework records total wall-clock latency and tokens per second (completion tokens divided by total latency). These latency metrics are reported as min, max, mean, p50, and p95. Time-to-first-token is always identical to total latency because RunCompletion makes a single synchronous request and response through the gateway—there is no streaming involved, so no earlier moment exists to measure against. The field is preserved in the schema to keep the report JSON stable, so readers should expect TTFT and latency columns to read identically across every row.

Quality scoring uses an LLM judge that evaluates model output against the source transcript in a reference-free manner. Summarization and information extraction have no gold summary or annotated entity list to compare against, so the judge operates without reference answers. Text correction follows the same pattern: the candidate model corrects missed_text (a simulated corrupted transcript), and one reference-free judge call grades the output against missed_text on dimensions of faithfulness and improvement. The original transcript included in the dataset is not used for scoring—it was found not to be an accurate answer key, so any word-level distance calculation or reference-based judge would score the dataset's own errors as much as the model's. Reports keep those secondary scores, but the quality_score field is built from the two judge dimensions only; re-running text_correction produces a score that stands on its own.

Statistics are broken down by audio-length bucket rather than reported as a single per-model number. Buckets such as 30s, 120s, 900s, 3600s, and others are read off each record—whatever is actually present in the loaded data. Buckets are never hardcoded, so if the normalization side adds a new bucket later, the framework adapts without code changes. This bucketed approach is the whole point: a model that performs well on 30-second calls but fails on hour-long ones is fundamentally different from a model that is merely "fast and accurate" in a summary that collapses all lengths.

Dictation runs as included task alongside the other three, flowing through the same staging and canonical-merge pipeline, but is loaded and scored differently. Each dictation record carries an identical system prompt (prompts.dictationSpec.Task) describing the spoken-command grammar under test—commands like "comma" produce commas, "all caps " uppercases the word, and "new paragraph" starts a new paragraph. Dictation records come from normalized/dictation/*.jsonl files (e.g. synthetic.jsonl), never from aai_transcripts.jsonl—the schema and filename differ. Each record also carries an input.instruction field, but it is deliberately never read; the single shared prompt is the only one ever sent to the model regardless of what that field contains.

Dictation grading is entirely mechanical. Every record carries a real answer key (expected.truth_text), so ScoreDictation computes exact_match, word_error_rate, punctuation_accuracy, capitalization_accuracy, and formatting_command_accuracy directly against truth_text in code, with zero LLM calls. Because these scores are mechanical, dictation's five score names are percentages (0-100, except exact_match which is a plain 0/1) rather than a 1-5 Likert rubric. These scores fold into the same quality_score/task_quality breakdown that every other task uses; the normalization code simply inverts word_error_rate (since lower is better) and scales the remaining dimensions.

The standard benchmark's RunCompletion always has TimeToFirstToken identical to TotalLatency. Every run additionally includes a separate pass that measures real time-to-first-token via streaming (RunCompletionStream), but only for the 30s and 120s audio-length buckets. TTFT here is a real-time-adjacent responsiveness signal, not the full transcript-size sweep the standard benchmark produces. The streaming pass never judges—there is no LLM-judge call or schema check; a TTFT sample only measures how fast the first token returns, never what the model said.

The two passes—standard benchmark and TTFT streaming—run sequentially, never concurrently. Running them together would send both streaming and synchronous requests to the same candidate models at once, compounding against the gateway's per-account-per-model rate limit for both passes' numbers, not just one. The TTFT pass has its own sample types, cells, report types, and staging directories (results_ttft/ and errors_ttft/), which mirror but never share with the standard results/ and errors/ directories. A run's TTFT directory is named like its standard counterpart (results_ttft/ beside results/), so one name finds everything the run staged.

TTFT reports merge into the summary: a TTFT latency breakdown (ttft_n through ttft_max) appears in every summary row by (model, bucket)—new columns alongside existing ones, blank for any bucket the TTFT pass never measured (anything other than 30s or 120s).

Canonical result accumulation lives in results/canonical//, where version folder names use dashes (e.g. 0.1.2 becomes 0-1-2). This is a single running result set that accumulates coverage across however many separate runs it took to benchmark every model. It differs from results/, which holds fresh, uniquely-named subfolders per run. Appending to canonical is keyed by (model, reasoning effort, task, bucket): new data for a combination entirely replaces old data for that same combination, however many records the old run sampled. Every combination the new run didn't touch passes through byte-for-byte unchanged—for instance, a run restricted to -buckets 3600s only replaces that bucket's rows, leaving 30s, 120s, and 900s's already-merged data alone. A naive recompute-on-every-append is avoided for this reason.

Every run executes the standard benchmark and the TTFT pass back-to-back sequentially (never concurrently, again for the rate-limit reason). It stages both under results/ and results_ttft/, prints a combined review summary for human sanity-checking, then either merges into canonical immediately with -approve-merge (for scripted/CI use) or asks for interactive confirmation on stdin first—the same plan-then-confirm shape as terraform apply. Declining leaves both staged directories on disk untouched. A successful merge re-runs the summary code over the canonical version directory afterward, so its summary.json and summary.csv reflect the merge immediately.

Benchmarking commands specify models and tasks, for example: go run . -models new-model-1,new-model-2. Every model call costs money, and the judge step is the part most likely to be reworked—a new rubric, a different judge model, or a judge that was down. Redoing judging should not mean paying for model calls again. Every call that comes back is written to disk the moment it returns, before it is judged, in the run's staged directory beside its report. The TTFT pass's streaming calls are stored the same way under results_ttft/. Because files are written immediately, a judge that is down or a run killed partway through still leaves every output already paid for. The -replay flag re-runs the judge step over earlier run outputs without calling models again.

-replay takes an earlier run's name (its directory name under results/ as logged at the end of that run) or a path to it, along with -models, -tasks, -buckets, -max-samples-per-bucket, and -reasoning-effort as for any run. For every (model, record) the earlier run stashed, both passes skip the model call: the standard pass uses stashed output (and the judge still runs live over it), while the TTFT pass uses the stashed measurement. Original call latency and token counts are kept so a replayed sample reports the numbers it was measured at; cost is stamped from the model's current pricing. Anything the earlier run didn't stash runs live—as in a normal run: a failed call, a model not included, a record beyond -max-samples-per-bucket, or a pass that never finished. A replay also finishes a run that was interrupted and adds a model to it. The log reports per task and pass how much came out of the stash and how much was called live.

Before anything is called, -replay refuses what can't be replayed faithfully: a name that isn't a run under results/, or a run that stashed nothing (only a run made since outputs were stashed can be replayed, and a mistyped name would otherwise run everything live at full price); a run that stashed none of what this run asks for (none of its models at the reasoning effort it will call them at, for none of its tasks); or a requested task whose prompt has changed since the run stashed its outputs, since replaying would score old prompt outputs as the new one's. A run that stashed only one pass (e.g. TTFT pass never finished) is replayed for that pass with a warning, and the other runs live.

-skip-judge makes every model call (both passes) and stashes the output, but judges nothing—neither the LLM judge nor dictation's mechanical scoring—and ends before the canonical merge, since unjudged samples would replace canonical's judged rows for the same (model, task, bucket). The staged report has samples with no scores. The run can be judged later with -replay , which makes no model calls.

Every summary row also carries a task_quality breakdown alongside the combined quality_score. One more 1-100 score per task contributes to that (model, bucket) row, computed the same way quality_score is but restricted to just that task's own samples. Summarization's score is never diluted by text_correction's or information_extraction's. The row's quality_score is the equal-weight mean of these per-task scores: every task counts the same regardless of how many samples it ran, and a task with no judged samples (not run at all) is left out of the mean rather than counted as zero. In the CSV, this appears as one quality_ column per task actually present in the data—discovered, not hardcoded.