Parsing Streams of JSON Documents at Over a Hundred Gigabytes per Second on CPUs

vendredi, 13 nov. 2026·
Jaël Champagne Gareau
Jaël Champagne Gareau
,
Daniel Lemire
Résumé
Large volumes of data are shipped as streams of JSON documents, one per line, and ingesting them is often a bottleneck for analytics. We present a parallel parser for such streams: workers claim byte ranges from one atomic counter and snap them to delimiters that cannot occur inside a value, so a document is never split and no lock is taken on the hot path. Because the parser is on-demand, it materializes only the fields a query names, which fuses query evaluation into the parse. We evaluate the design on five Intel and AMD servers against Pison, a state-of-the-art structural-index parser, and against the tree-building parsers in common use, verifying identical results throughout. With dynamic assignment the best rate per machine ranges from 105 to 229 GiB/s (up to 246 GiB/s with static assignment), several times the throughput that graphics processing unit (GPU) parsers report on the same datasets. Against Pison we are 4.3–17.0$\times$ faster end to end, because its index requires a padded copy that its published timings exclude. Against yyjson, the fastest tree-building parser, we retire fewer instructions per byte on every dataset. We also compare with three data systems—Apache Arrow, DuckDB and Polars—and with cuJSON on its own end-to-end workload. Our parser and benchmark are open source.
Type
Publication
The VLDB Journal
publication

Submitted version, under review at The VLDB Journal.

Jaël Champagne Gareau
Auteurs
Chercheur postdoctoral en informatique
Je suis actuellement chercheur postdoctoral en informatique à l’Université TÉLUQ, où mes travaux portent sur l’accélération de la conversion de nombres entiers et flottants en chaînes de caractères décimales. Au cours de mon doctorat, j’ai conçu des algorithmes et des structures de données exploitant l’architecture moderne des ordinateurs afin de résoudre de grandes instances de processus décisionnels de Markov (MDP). Durant ma maîtrise, j’ai développé des algorithmes de planification d’itinéraires pour véhicules électriques, visant à déterminer le chemin optimal entre deux points tout en minimisant le temps total du trajet (déplacement, recharge et attente aux bornes).

Citation