Parsing Streams of JSON Documents at Over a Hundred Gigabytes per Second on CPUs
vendredi, 13 nov. 2026·
,
Jaël Champagne Gareau
Daniel Lemire
Résumé
Large volumes of data are shipped as streams of JSON documents, one per line,
and ingesting them is often a bottleneck for analytics. We present a parallel
parser for such streams: workers claim byte ranges from one atomic counter and
snap them to delimiters that cannot occur inside a value, so a document is
never split and no lock is taken on the hot path. Because the parser is
on-demand, it materializes only the fields a query names, which fuses query
evaluation into the parse. We evaluate the design on five Intel and AMD servers
against Pison, a state-of-the-art structural-index parser, and against the
tree-building parsers in common use, verifying identical results throughout.
With dynamic assignment the best rate per machine ranges from 105 to
229 GiB/s (up to 246 GiB/s with static assignment), several times the
throughput that graphics processing unit (GPU) parsers report on the same
datasets. Against Pison we are 4.3–17.0$\times$ faster end to end, because its
index requires a padded copy that its published timings exclude. Against
yyjson, the fastest tree-building parser, we retire fewer instructions per byte
on every dataset. We also compare with three data systems—Apache Arrow,
DuckDB and Polars—and with cuJSON on its own end-to-end workload. Our parser
and benchmark are open source.
Type
Publication
The VLDB Journal
Submitted version, under review at The VLDB Journal.

Auteurs
Chercheur postdoctoral en informatique
Je suis actuellement chercheur postdoctoral en informatique à l’Université
TÉLUQ, où mes travaux portent sur l’accélération de la conversion de nombres
entiers et flottants en chaînes de caractères décimales. Au cours de mon
doctorat, j’ai conçu des algorithmes et des structures de données exploitant
l’architecture moderne des ordinateurs afin de résoudre de grandes instances
de processus décisionnels de Markov (MDP). Durant ma maîtrise, j’ai développé
des algorithmes de planification d’itinéraires pour véhicules électriques,
visant à déterminer le chemin optimal entre deux points tout en minimisant
le temps total du trajet (déplacement, recharge et attente aux bornes).