Automatic Music Transcription for guitar has long been stuck on the wrong side of the difficulty curve compared to piano. Piano AMT hit high note-level accuracy years ago through CRNN models and large paired datasets, but guitar transcription still struggles with three problems that no single system has solved together: expressive techniques like slides, bends, and harmonics are ignored or mislabeled, the same pitch can be played at multiple string-fret positions and models pick the wrong one, and systems trained on clean studio recordings fall apart on noisy real-world audio. TART, a four-stage modular pipeline from Gupta, Park, Guzman, Gowda, Konduri, Lian, Netzorg, and Anumanchipalli at UC Berkeley, tackles all three at once and produces guitar tablature with both fingering and expressive technique annotations directly from audio. It was presented at ISMIR 2026.

Why guitar transcription is harder than piano

Piano has 88 keys and a fixed mapping between pitch and key. Guitar has six strings and a fretboard, which means the same MIDI pitch can be produced at multiple string-fret combinations. A note at pitch 64 (E4) can be played on the 1st string at fret 0, the 2nd string at fret 5, the 3rd string at fret 9, the 4th string at fret 14, or the 5th string at fret 19. Correct tablature requires picking the combination a guitarist would actually use, which depends on what notes come before and after, the player's hand position, and timbral differences between strings. Prior systems either ignored this problem entirely or treated it as a pure symbolic search, missing the acoustic cues that distinguish the same pitch played on different strings.

On top of that, guitarists use a vocabulary of expressive techniques that piano simply does not have. Slides, bends, vibrato, hammer-ons, pull-offs, harmonics, palm muting, and percussive hits each modify the sound in ways that matter for transcription. No prior system has produced a unified taxonomy covering all of these. Some classifiers handle bends and slides but ignore harmonics and percussion. Others focus on percussive techniques but skip slides entirely. The label schemes across datasets are incompatible, making it hard to train a single model that covers everything.

A four-stage pipeline, not a single end-to-end model

TART's core design decision is modularity. Rather than training one massive model to do everything, the authors break the problem into four stages, each of which can be independently trained, evaluated, and improved. Stage 1 transcribes audio to MIDI. Stage 2 classifies expressive techniques for each note. Stage 3 assigns string and fret positions. Stage 4 merges everything into a beat-aligned MusicXML tablature score. This architecture lets each stage use the architecture best suited to its subtask, and lets the pipeline inherit improvements from any individual stage without retraining the whole system.

Stage 1: Noise-robust audio-to-MIDI transcription

The first stage uses the high-resolution CRNN from Kong et al., which predicts onset confidence, offset confidence, frame activation, and velocity from a log-mel spectrogram at 100 frames per second. The authors leave the architecture unchanged and focus on making it generalize to real-world recording conditions.

The training strategy is the main contribution here. They pool four datasets that span different instrument domains: GAPS (classical guitar), Guitar-TECHS (electric guitar with diverse hardware), a Francois Leduc acoustic dataset, and the DI subset of GOAT. This domain diversity is critical because models trained on clean acoustic recordings fail on electric pickups and distortion. They fine-tune the CRNN on 30-second segments with a 10-second hop, decaying learning rate by 0.9 every 10,000 iterations.

They also design a stochastic noise augmentor specifically for guitar. The standard piano augmentation pipeline from Kong et al. includes reverb and aggressive equalization, both of which are destructive for guitar: reverb smears the attack transients that onset detectors rely on, and EQ removes harmonics critical for pitch tracking. Instead, TART applies a zero-phase 80 Hz high-pass filter, then mixes in white noise, pink noise, or 60 Hz hum at a random SNR between 25 and 45 dB, with 50% probability per batch. This preserves onset alignment while simulating realistic recording conditions.

To evaluate generalization, the authors create two new benchmarks: Noisy GuitarSet and Noisy EGDB. They take the original clean recordings, apply the same noise augmentation, and additionally convolve with a room impulse response from the EchoThief collection (sampled from real spaces like nature recordings, sanctuaries, and venues), then truncate to the original length to preserve frame-level alignment. These simulate recordings made with phones, laptops, and low-quality microphones in diverse environments.

The results tell a clear story. On clean GuitarSet, the Riley et al. model narrowly leads at 88.1% F50 versus TART's 87.4%. But Riley's model collapses on EGDB (68.9% vs. TART's 79.0%) and on the noisy benchmarks. TART's average across all four benchmarks is 81.35% F50, beating the next-best system by 6.67 points. The margin is largest on EGDB (+10.1 points) and Noisy EGDB (+9.3 points), showing that the multi-dataset training and noise augmentation produce genuine robustness rather than overfitting to one dataset's characteristics.

Stage 2: A unified technique classifier with 160K parameters

The second stage takes each note from Stage 1 and assigns one of nine technique labels: bend, hammer-on/pull-off, harmonics, kick drum, palm muting, picking (no technique), slide, snare drum, and vibrato. No single public dataset covers all nine techniques with compatible labels, so the authors consolidate five datasets (IDMT-SMT-Chords, Guitar-TECHS, AGPT, EG-IPT, and Magcil) into a single unified training set with a common taxonomy.

The classifier architecture is a temporal CNN-BiLSTM. For each note, the system extracts the audio chunk from onset to offset and computes a feature sequence at 23 ms hop: 40 MFCCs, 40 log-mel bands, and 12 chroma coefficients stacked into a 92-dimensional vector, z-normalized per feature across time. Sequences are padded or truncated to 128 frames (about 3 seconds), yielding a uniform 128x92 input tensor.

The network has two 1D convolutional blocks (64 and 128 filters, kernel size 3), each followed by batch normalization, max pooling, and dropout at 0.3. These feed into a bidirectional LSTM with 64 units per direction (128 total), then a fully connected head with 128 units, ReLU, batch normalization, dropout, and a softmax over the nine classes. The total parameter count is 160,000.

The results on the unified test set are strong. Macro F1 is 95.9%, with all classes exceeding 95% recall. Percussion techniques (kick drum, snare drum) reach 99.5% recall. The hardest class is vibrato at 89.4% F1, which gets confused with bend because both involve pitch modulation. The comparison with prior work is striking: Fiorini et al.'s convolutional network scores 62.7% macro F1 with 3.7 million parameters, and Stefani et al.'s dense MLP scores 71.6% with 2.08 million parameters. TART's CNN-BiLSTM achieves 95.9% with 13x fewer parameters than Stefani et al., demonstrating that modeling temporal dependencies matters more than scaling feedforward capacity for this task.

Stage 3: AudioFret, the audio-conditioned string-fret assigner

This is where the paper's most novel contribution lives. The string-fret assignment problem is fundamentally ambiguous: the same pitch can be played at multiple positions, and purely symbolic models (like the Fretting-Transformer) cannot distinguish them because they only see MIDI tokens. TART's AudioFret model injects per-note timbral features from the raw audio alongside the symbolic MIDI sequence, allowing the model to exploit the fact that the same pitch played on different strings produces audibly different spectra due to differences in string gauge, tension, and overtone structure.

The architecture builds on the Fretting-Transformer, a T5-style encoder-decoder with a unified vocabulary for MIDI and tablature tokens. The authors scale the backbone from the original smaller configuration (d_model=128, 3 layers) to a larger variant (d_model=256, 6 layers, 8 heads, about 15 million parameters) with gated-GELU feed-forward layers. For each input note, they extract a 200 ms mel-spectrogram around its onset, pass it through a lightweight CNN (three convolutional blocks with batch normalization and ReLU), and prepend the resulting per-note audio embeddings as a contiguous block immediately before the MIDI token sequence. The T5-style self-attention fuses audio and symbolic information end-to-end without explicit fusion hyperparameters.

Training proceeds in two phases. First, the scaled T5 backbone is pre-trained from scratch on SynthTab and DadaGP (17,255 training tracks), augmented with capo positions 0-7 and four tuning variants (standard, half-step down, full-step down, drop D) as conditioning tokens. Second, the CNN audio encoder is pre-trained as a string classifier on GAPS, GOAT DI, and Guitar-TECHS, then the entire AudioFret model is jointly fine-tuned end-to-end for 30 epochs.

At inference, beam search (beam width 4) replaces greedy decoding, with two constraints: every note must be followed by a valid duration token, and no chord may require a fret span greater than 5 (the realistic reach limit of a guitarist's hand).

The ablation results in Table 4 isolate where the gains come from. Scaling the symbolic backbone from the original Fretting-Transformer raises average Tab F1 from 63.3% to 67.0%, a 3.7 point gain with no audio information. Adding audio conditioning on top yields another 4.8 points, for a total of +8.5 points over the original baseline. AudioFret achieves 71.8% average Tab F1 with 100% pitch accuracy. The Audio-only ablation (just the CNN string classifier, no symbolic decoder) scores 58.4%, and the Symbolic-only ablation (scaled T5 with no audio) scores 67.0%, confirming that both modalities contribute.

Stage 4: From annotations to playable tablature

The final stage merges the parallel outputs of Stages 2 and 3 (technique labels and string-fret assignments) with a tempo estimate from BeatNet into a single event stream stored in a JAMS annotation file. It then renders this into a beat-aligned MusicXML tablature score.

The conversion involves two steps. First, rhythmic quantization: onsets are clustered within 30 ms to remove micro-timing errors, and both onsets and durations are quantized to a 1/16-note grid using BeatNet's estimated tempo. Notes sharing the same quantized onset are grouped into chords, and same-string collisions are resolved by retaining the longer note. Second, score rendering: the quantized event stream is exported to MusicXML, with single-note techniques (bend, harmonic, vibrato, palm muting, kick drum, snare drum, picking) written directly, and two-note techniques (hammer-on/pull-off, slide) inferred by pairing consecutive notes on the same string within a 1-beat window.

Error propagation through the pipeline

The modular design has a cost. Because Stages 2 and 3 depend on Stage 1's MIDI output, errors in Stage 1 propagate downstream. Table 5 quantifies this: the oracle Tab F1 (using ground-truth MIDI as input to AudioFret) averages 71.83%, but the end-to-end Tab F1 (using Stage 1's predicted MIDI) drops to 54.08%, a propagation cost of 17.75 percentage points. This is the pipeline's most significant weakness. Every improvement to Stage 1's MIDI accuracy will directly improve the final tablature, which makes the modular design a strength for incremental improvement: you can swap in a better transcriber without retraining the rest of the system.

Limitations the authors acknowledge

The audio-to-MIDI stage does not detect unpitched notes like percussion, which prevents downstream technique annotation for percussive hits. The technique classifier assigns only one technique per note, so simultaneous techniques like a bend performed with vibrato cannot be represented. The tablature generation uses fixed-rhythmic quantization for chord grouping, which can produce occasional misgroupings. And the propagation cost of 17.75 points means the end-to-end system loses significant accuracy from Stage 1 errors alone.

The evaluation is zero-shot, meaning no fine-tuning on the test datasets, but the training data consolidation across five technique datasets and four transcription datasets is non-trivial work that may not be reproducible by teams without access to all five sources. The noisy benchmarks are synthetic corruptions rather than actual field recordings, so they test one axis of real-world difficulty (noise and room acoustics) but not others (variable playing quality, multi-instrument mixtures, or background music).

What this means for developers building music tools

TART demonstrates that modularity beats end-to-end when the subtasks are well-defined and the intermediate representations are clean. A developer building a guitar learning app, a transcription service, or a music production tool could adopt individual stages rather than the whole pipeline. The technique classifier alone, at 160K parameters, is small enough for real-time use on mobile hardware and outperforms prior work by a wide margin. The AudioFret string-fret assigner shows that timbral cues from raw audio contain real information that symbolic-only models miss, a finding that applies to any pitch-redundant instrument transcription task.

The pipeline also points to a practical deployment pattern: run Stage 1 on-device for low-latency note detection, run Stages 2-4 server-side where more compute is available, and stream the annotated MIDI between them. The MusicXML output is a standard format that any notation software can import, making integration with existing tools straightforward.

The biggest open question is whether the propagation cost can be reduced. The 17.75-point gap between oracle and end-to-end performance suggests that better audio-to-MIDI transcription would yield the largest gains. Alternatively, a joint model that performs MIDI detection and string-fret assignment simultaneously could eliminate the cascading error, though at the cost of the modularity that makes TART easy to improve incrementally.

Read the paper on arXiv