How fast can you fix a UTF-16 string in C#?

C# strings are encoded as UTF-16, meaning most characters occupy a single 16-bit code unit while characters outside the basic multilingual plane, including emoji, require a high surrogate followed by a low surrogate. When a surrogate appears without its proper partner, the result is an ill-formed string. Sending such strings to disk or over the network is a well-known bad practice, but detecting and repairing them efficiently has historically been expensive, especially in high-throughput systems.

The Problem and the JavaScript Precedent

JavaScript already provides two built-in functions for this problem: String.prototype.toWellFormed() replaces every lone surrogate with the replacement character U+FFFD, and isWellFormed() reports whether any repair is needed. The approach is straightforward but hides a performance question: how fast can you scan a string and find the broken pairs?

Daniel Lemire has now brought both functions to C# through his SimdUnicode library, via pull request 54. The algorithm matches what was contributed to the V8 JavaScript engine, so Chrome already performs these repairs the same way. In C#, the usage is simple: UTF16.ToWellFormed(input) returns the original string instance when the input is already well-formed, meaning zero allocation in the common case. UTF16.IsWellFormed(span) returns a boolean indicating whether any lone surrogates exist.

Sequential Scanning vs. SIMD Parallelism

The conventional approach in C# scans through the string one code unit at a time, searching for any character in the surrogate range and then checking whether the following character forms a valid pair. For each error found, it writes U+FFFD at that position and continues. This works, but the sequential scan becomes a bottleneck when processing large buffers of text.

SIMD instructions change the equation by allowing a processor to compare multiple values simultaneously. Recent x64 chips from both AMD and Intel offer significantly better data parallelism than ARM processors, though ARM has its own powerful instruction set. The SimdUnicode library leverages these instructions to scan for ill-formed surrogates across many code units at once rather than stepping through them one by one.

The Performance Numbers

On an Intel Xeon Gold 6548N (Emerald Rapids) with AVX-512 and .NET SDK 10, the results are striking. Latin text validation runs at 69 GB/s using the SIMD approach compared to 33 GB/s with the sequential IndexOfAnyInRange method, roughly a factor of two improvement. But the real divergence appears with emoji-heavy input.

Emoji strings consist entirely of surrogate pairs and are well-formed, meaning the sequential scanner must examine every pair to confirm there are no lone surrogates. Under that workload, the conventional search drops to 0.4 GB/s, while the SIMD-based check maintains 53 GB/s. The gap is enormous because the sequential method pays full cost for every character even when nothing is wrong.

There is a further optimization: when the input is already well-formed, ToWellFormed simply returns the original string without any allocation or copying. This makes it effectively a zero-cost operation for clean input, as fast as a pointer return.

When the buffer form does need to write replacements, every code unit must be copied or replaced, making it effectively a memory copy operation. On well-formed input, the SIMD implementation runs at roughly the speed of a plain memory copy. The same tests on an Apple M4 Max showed similar results, though with slightly narrower margins compared to the Intel chip.

Why This Matters

Unicode correctness is not a corner case for modern applications. Text arrives from many sources, often in imperfect states, and systems that process it at scale need to handle malformed input without becoming a bottleneck. The fact that a well-formed string check can now run at over 60 GB/s on modern hardware means that correctness and performance are no longer in tension for C# developers working with text at volume.

The underlying work has also been published academically. Clausecker and Lemire documented the approach in a 2026 paper titled "Fixing ill-formed UTF-16 strings with SIMD instructions" in Software: Practice and Experience, available on arXiv. The code is in SimdUnicode, and the algorithm is already shipping inside Chrome via V8.