Procedural Memory That Gets Better Without Retraining the Model
Professional graphic design sits at an uncomfortable intersection for AI agents. A single design brief can require dozens of interdependent operations, arranging assets, manipulating typography and vectors, building masks and effects, and revising earlier decisions while preserving editability. Unlike code, where a test suite can tell you whether the output is correct, design has no reliable programmatic oracle. A "double exposure" composition that looks technically fine might violate a brand guideline, or a logo that satisfies every explicit requirement might still clash aesthetically. The feedback is noisy, partial, and often subjective.
This makes graphic design a worst-case scenario for agent improvement. Gradient-based training needs clean reward signals. Reinforcement learning from user traffic needs credit assignment across long action sequences. And when the backbone is a hosted API model you cannot fine-tune, the question shifts from "how do we train the model" to "how do we train everything around the model."
Designer-RSI, from researchers at Adobe and Brown University, proposes a concrete answer: treat an external procedural memory, a library of natural-language skills, as the object that evolves, while keeping the foundation model frozen. The model weights never change. The tools never change. What changes is a curated collection of SKILL.md playbooks that the agent retrieves and follows at execution time.
Why Existing Approaches Fall Short
The standard playbook for improving agentic systems is one of two things. Either you fine-tune the model on task-specific data, which requires access to weights, clean supervision, and compute budget for training. Or you improve prompting, which is fast but brittle: a few-shot prompt or a system message can steer behavior on easy cases but does not transfer across tools, does not adapt over time, and does not compound across rounds.
Self-improving agent research has explored reflection loops where the agent critiques its own outputs and rewrites its instructions. This works in narrow domains with verifiable rewards (code generation with test suites), but breaks down when feedback is noisy or absent. The agent can hallucinate improvements, amplify its own biases, or overfit to the grading heuristic.
Designer-RSI sidesteps both problems. The backbone model stays frozen, avoiding the cost and complexity of weight updates. But the procedural memory is not static prompting. It is a persistent, structured, evolving artifact that accumulates reusable design procedures from real user traffic, with a conservative admission mechanism that prevents bad changes from shipping.
The System Architecture
The core agent is a frozen frontier language model controlling equivalents of Adobe Photoshop, Illustrator, and InDesign through more than 230 tools. Each tool maps to a specific design operation: creating layers, applying masks, importing assets, adjusting typography, rendering intermediate states. The agent receives a brief (a natural-language request describing the artifact, its concrete requirements, and its stylistic goals), retrieves relevant skills from the bank, selects a subset of tools, and executes a chain of operations through intermediate document states to produce a final rendered image.
The skill bank is a collection of SKILL.md files. Each file is a natural-language playbook describing a reusable design procedure: what subtasks it covers, which tools to call in what order, common failure modes to avoid, and conditional branches. A procedure sits between an atomic tool call and an entire trajectory, specific enough to guide execution, general enough to transfer across briefs.
Before execution, a retrieval step queries the skill bank using the brief and the tool list, selecting the top-k most relevant skills (k=3 by default) and injecting their playbooks into the model context. The runtime also progressively discloses a reduced tool set matching the retrieved skills, so the model does not need to navigate the full 230-tool catalog.
The four runtime roles are: a Prompter that poses design briefs (from real user data or LLM-generated variants), a Solver (the agent itself), a Grader (a multimodal evaluator that scores each image and explains why unmet requirements failed), and a Reflector that converts failures into targeted edits of SKILL.md files. Only the SKILL.md files change. Everything else is fixed.
How Widening Works: Discovering Missing Skills
Widening identifies recurring subtasks that the current skill library does not cover. For each trajectory, a frozen LLM extracts and canonicalizes the subtasks actually performed. A subtask counts as uncovered if no retrieved skill addresses it, either because retrieval returned nothing or because the retrieved skills cover a different part of the task. Subtasks associated with a skill blamed for a poor outcome also count as uncovered.
Uncovered subtasks accumulate in a persistent coverage pool keyed by canonical label. Once a label reaches k_min = 3 occurrences across the traffic, a frozen LLM distills those cases into a candidate skill. The candidate is admitted only if it passes the replay gate against the no-skill baseline. If rejected, the candidate is discarded but its occurrences remain in the pool, allowing further evidence to accumulate across evolution rounds.
The threshold k_min = 3 is deliberately conservative. A skill minted from a single failure could be an artifact of bad asset retrieval or a one-off edge case. Requiring three occurrences ensures the gap is systemic, not incidental. The authors found that minted skills are not redundant with the seed bank: their nearest-neighbor embedding distance to the cold-start skills exceeds the seed bank's internal spacing (0.215 vs. 0.158 median cosine distance, Mann-Whitney p < 10^-8), confirming that widening covers genuinely new intents.
How Deepening Works: Hardening Existing Skills
Deepening revises existing skills that repeatedly produce failures. The mechanism is asymmetric by design: selection is cheap and permissive, while the replay gate decides what ships.
Each trajectory records which skills it retrieved. A trajectory scoring below the success threshold (s_j < 0.6) counts as a failure against every skill it retrieved. The Reflector selects every skill whose failure count meets a threshold (default 2), most-failing first. This prioritization is a low-cost heuristic, not the quality filter.
For each selected skill, the Reflector receives the brief, the per-requirement outcomes and "why-bad" rationales, the current SKILL.md, and a contrastive set of this same skill's successful calls on similar tasks, represented by the tool-call sequences, intermediate waypoint results, and thinking tokens that actually worked. The successful runs serve as the do-not-regress baseline, while the failure rationales identify what needs improvement. The Reflector reasons over a concrete success-failure divergence rather than from failure text alone.
The Reflector emits a targeted edit naming the section and the change. When repeated targeted rewrites of the same skill have failed the gate, it escalates to a major rewrite of the whole skill. Rejected rewrites trigger an optional exploration cycle that probes the skill's failing prompts with and without skills and distills toward whichever arm succeeded, adjudicating the skill's fate as update, delete, or keep.
The Replay Gate: Conservative Admission Under Noisy Feedback
Both widening and deepening propose changes. A single replay gate decides which ones ship. The gate addresses two sources of confounding that make naive improvement detection unreliable.
First, the VLM Grader's absolute score for the same image drifts across runs. "Accept if the mean score rose" can confuse judge drift with improvement and admit regressions. The gate therefore never uses absolute scores. Second, outcomes depend on more than the skill: asset retrieval and other upstream state can differ between arms, allowing a candidate to win simply because it received better inputs.
The gate uses matched replay. For each proposed change, it samples prompts that exercise the skill and generates several contexts per prompt, each with distinct retrieved assets and upstream state. Each context is then frozen and replayed fresh in the same batch under both arms: the candidate versus the incumbent for a rewrite, or versus the no-skill agent for a mint. The resulting outputs are judged pairwise under order randomization, so within each context the only difference under test is the skill condition. A prompt is won only if the candidate wins a majority of its contexts.
A change ships only if it loses no prompt and wins at least one:
(∃ prompt won) ∧ (¬∃ prompt lost)This is an asymmetric criterion. It is easy to reject a change: any single regression kills it. It is hard to accept one: the change must win on at least one prompt without ever losing. This asymmetry reflects the operational reality that regressions in production are far more costly than missed improvements.
The Evolution Loop in Practice
The system runs for five rounds over 1,406 non-overlapping briefs drawn from real user traffic and LLM-augmented variants. Each round replays briefs through the agent to produce graded trajectories, which drive widening and deepening. Across five rounds, this yields 1,869 graded trajectories without human labels.
The dynamics follow a predictable pattern. Round 1 is dominated by repair: the seed bank was derived from internal documentation and has many skills that work in theory but not in practice. 39 of 59 rewrites and 44 of 101 mints pass the gate. Widening lags because gaps must recur across k_min requests before minting.
Minting peaks in rounds 2 and 3 (22/46 and 26/40 committed), growing the bank from 77 to 124. Round 4 mints heavily but many skills are v1, never revised, so they misfire on requests outside their originating cluster. Round 5 reverses the mix, committing 21 rewrites and only 4 mints, becoming the strongest round at every threshold. The bank ends at 139 skills.
The trajectory is not monotonic. Round 4 falls below no-skill at completeness thresholds below 0.3, while retaining gains at the high end. This is the coverage-reliability trade-off in action: minting expands coverage but initially lacks multi-trial verification, while rewriting converts coverage into reliability. The two mechanisms are complementary, not redundant.
What the Numbers Show
On the internal benchmark (200 held-out human-authored briefs, fixed across all rounds, disjoint from evolution data), the evolved skill bank raises performance at essentially every completeness threshold. Relative to no skill, round 5 raises the share of briefs at completeness >= 0.5 from 86% to 93%, at >= 0.9 from 43% to 56%, and at exactly 1.0 from 24% to 32%. The largest improvement is +13 percentage points at the 0.9 threshold.
On general text-to-image benchmarks, the results are strong across multiple frozen backbones. Claude-Sonnet-4 sees GenEval2 execution success jump from 72.7% to 99.3%, with generation quality rising +11.99 points. DPG-Bench success rises from 82.7% to 100%. The average quality improvement across all four general benchmarks is +7.70 points for Sonnet-4 and +9.67 for Qwen3.6-27B.
On specialized graphic design benchmarks judged pairwise by GPT-5.4 in blind, two-order comparisons, the Evolve agent achieves 61.8% and 67.6% overall win rates against the no-skill agent on Claude-Sonnet-4 and Claude-Opus-4.6 respectively. The margins are largest on CreatiDesign (68.1% and 71.7%) and OpenCOLE (66.0% and 64.0%), which involve multi-step planning and layout constraints.
The ablation study isolates the two mechanisms. On 200 held-out briefs, widening alone reaches 49.4% win rate, deepening alone reaches 48.6%, but their combination reaches 58.5% (p = 0.025). The superadditive gain (+3.85 completeness points over cold start) reflects loop coupling: minted skills require refined retrieval descriptions to surface, while rewriting reroutes unfixable failures to the gap store for future minting.
Latency and Cost
Skill retrieval adds roughly 28% prompt tokens over the base agent. But evolution adds no marginal token cost. With top-k (k=3) matching, Evolve actually uses fewer prompt and output tokens than the cold-start bank (436.6k/5167 vs. 445.2k/5298), consistent with more direct execution and fewer corrective retries. Mean wall-clock latency overhead is 3.4% to 6.2% across backbones, and in some cases Evolve is faster (113 to 82 seconds on DPG-Bench for Sonnet-4), because the skill-guided agent makes fewer wrong turns.
Where the Approach Hits Its Limits
The authors are candid about what this approach cannot do. A natural-language skill can describe a preferred procedure, but retrieval alone may not make the model follow it when the model has a strong default strategy. The model may ignore the playbook and fall back to its training distribution. Fine geometric operations remain limited by both model perception and automated verification: sub-pixel alignment, precise spacing, and color-accurate grading are hard for VLM-based evaluators.
Long procedures lose fidelity as instructions accumulate over many execution steps. A 30-step procedure is harder to follow reliably than a 5-step one, and the model may skip or reorder steps despite the playbook. The replay gate is also local to the evaluated cases: rejecting observed regressions on the replay set does not guarantee monotonic improvement over the full user-traffic distribution.
There is also a cold-start problem. The seed bank of 76 skills was distilled from internal documentation, not from traffic. If the documentation is incomplete or outdated, the initial coverage is limited, and widening requires three occurrences before minting, so early rounds are slow.
What This Means for Agent Builders
The practical lesson is that procedural memory is a viable alternative to fine-tuning for continual agent adaptation, especially when you cannot or do not want to update model weights. The approach works with hosted API models, requires no human reward labels, and compounds across rounds. The replay gate provides a principled way to filter changes under noisy, unverifiable feedback.
The two-mechanism design (widening plus deepening) is important. Coverage expansion without hardening produces skills that are broad but unreliable. Hardening without expansion produces skills that are reliable but narrow. The superadditive interaction means the whole is greater than the sum of its parts, and the gate keeps the system from drifting.
For developers building agentic systems in creative domains, the architecture is straightforward to replicate: a frozen model, a tool catalog, a skill bank of natural-language playbooks, retrieval at execution time, and an offline evolution loop with a conservative admission gate. The grading rubric (rendering, completeness, aesthetics, critique) is domain-specific but the framework is not. Any domain where feedback is noisy and long-horizon, and where you want to improve without retraining, can adapt this pattern.
Designer-RSI shows that the procedural memory surrounding a model can be the learning objective, not the model itself. That is a useful shift in perspective for anyone building agents that need to get better over time without touching the weights.
Read the paper on arXiv