How to Process Large JSON Files in Python With ijson

JSON is one of the most common formats for storing and exchanging data in Python applications. For small files, Python's built-in json module works perfectly well:

import json

with open("data.json") as f:
    data = json.load(f)

The problem starts when the file becomes large.

json.load() parses the entire JSON document and creates the corresponding Python objects in memory before you can start processing the data. A file that occupies a few hundred megabytes on disk can therefore require much more memory once converted into Python dictionaries, lists, strings, and numbers.

A better option for this type of workload is ijson, an iterative JSON parser that lets you process data incrementally. In this tutorial, we will use a large JSON file containing San Francisco land-parcel data. We will first process it with json.load(), measure what happens to memory, and then solve the same task with ijson.

Step 1: Downloading a Large JSON Dataset

For this experiment, we will use the San Francisco City Lots dataset. I am using Google Colab for this article. Download it using:

curl -L \
https://raw.githubusercontent.com/zemirco/sf-city-lots-json/master/citylots.json \
-o citylots.json

You can check its size with:

ls -lh citylots.json

The dataset is 181 MB in size, making it large enough to demonstrate the difference between loading and streaming JSON. The file follows the standard GeoJSON FeatureCollection structure. You can check it using this code:

import json

with open("citylots.json", "r") as f:
    data = json.load(f)

print(data.keys())
data["features"][0]

Output:

The important part for us is the features array.

Each feature represents one parcel and contains two main sections:

feature
├── properties
│   ├── BLKLOT
│   ├── BLOCK_NUM
│   ├── LOT_NUM
│   └── STREET
│
└── geometry
    └── coordinates

Our analysis only needs a few values under properties. However, if we use the standard JSON parser, Python still has to load all of the geometry coordinates as well.

The problem is this line:

Run: