Systems frequently log data as JSON one record per line NDJSON. GitHub publishes every public event through GH Archive roughly a gigabyte per hour. Suppose you want to know which repositories received the most stars over eight hours. That query spans 7.6 GB of JSON two point two million events.
The most straightforward Python code uses the json module reading each line and parsing it. That approach runs at 0.28 GB/s: 27 seconds for the eight-hour file. The orjson package is twice as fast. The msgspec package which decodes only the fields you declare is eight times as fast.
A server has many cores but a Python process with the GIL runs on one core at a time. The usual solution is a pool of processes: cut the file into pieces give a piece to each process and add up their counts. Each process returns a small table so little has to come back.
Python 3.14 has a free-threaded build python3.14t without the GIL threads run Python code simultaneously. The fastsimdjson library is threaded. You give it the data and a function. It parses the records on many threads and calls your function on the thread that parsed the record. You name the fields you want. It reads only what leads to them and your function receives those values not the whole record.
import collections
import fastsimdjson
def starred(event):
kind repo = event
if kind == "WatchEvent":
return repo
names = fastsimdjson.map_many(
starred data threads=192
fields=["/type"/repo/name"] drop_none=True)
stars = collections.Counter(names)
The variable data is the file mapped in memory. The two paths are the event type and the repository name. The drop_none parameter leaves out records that are not stars.
The test ran on an AWS instance c8a dot metal-48xl with two AMD EPYC 9R45 processors 192 cores using Python 3.14.10 orjson 3.13 and msgspec 0.22. The file is mapped before the clock starts. On one core fastsimdjson reads 8 GB/s three and a half times as fast as msgspec.
With 192 threads fastsimdjson counts the stars in the 7.6 GB in 55 milliseconds: 138 GB/s. The best pool of msgspec processes reaches 116 GB/s at 96 processes. More processes make it slower.
fastsimdjson is ahead at every worker count. The lead shrinks from three and a half times on one core to about 20%.
Counting is the best case for processes because little comes back. Often you want the values themselves. The author took six files of about a gigabyte from an existing benchmark Pison and pulled a field or two from every record. On the twitter file fastsimdjson reaches 277 GB/s. Four of the six files pass 100 GB/s. The other two are slower one stays near 20 GB/s still about four times as fast as msgspec and one is a tie.
Why the GIL matters for JSON parsing
Standard Python is limited by the global interpreter lock which prevents threads from running Python code concurrently. This means a single-process JSON parser cannot use all available CPU cores. The common workaround is multiprocessing splitting the input across separate processes and merging the lightweight results.
Free-threaded Python and threaded JSON parsing
Python 3.14 introduces a free-threaded variant that removes the GIL. On this build threads can execute Python code in parallel. The fastsimdjson library takes advantage of this by distributing record parsing across many threads. Each thread parses a subset of the input and the library invokes a user-provided callback on the thread that handled each record. The caller specifies which JSON paths to extract and the library reads only those subtrees.
Benchmark results across hardware and libraries
On a single core fastsimdjson achieves 8 GB/s roughly three and a half times the throughput of msgspec. When scaled to 192 threads fastsimdjson processes the 7.6 GB star-counting workload in 55 milliseconds delivering 138 GB/s. A multiprocessing setup using msgspec peaks at 116 GB/s with 96 worker processes; adding more processes actually reduces performance.
fastsimdjson leads at every concurrency level. The performance gap narrows as worker count increases because overhead from thread coordination and result merging becomes relatively larger.
Beyond star counting real-world field extraction
The star-counting scenario is favorable: each record contributes a tiny amount of output making aggregation cheap. In more typical usage the goal is to extract specific fields from every record. The author tested six approximately gigabyte files from the Pison benchmark pulling one or two fields from each JSON record.
On the Twitter dataset fastsimdjson sustains 277 GB/s. Four of the six files exceed 100 GB/s. The remaining two are slower one stabilizes near 20 GB/s still about four times faster than msgspec and one ties msgspec's speed.