26  Fetching JSON Data

If the data we want to fetch is in JSON format, we can use the requests package to fetch and process it.

Let’s consider this example "gradebook.json" file we have hosted on the Internet:

{
  "downloadDate": "2018-06-05",
  "professorId": 123,
  "students": [
    {"studentId": 1, "finalGrade": 76.7},
    {"studentId": 2, "finalGrade": 85.1},
    {"studentId": 3, "finalGrade": 50.3},
    {"studentId": 4, "finalGrade": 89.8},
    {"studentId": 5, "finalGrade": 97.4},
    {"studentId": 6, "finalGrade": 75.5},
    {"studentId": 7, "finalGrade": 87.2},
    {"studentId": 8, "finalGrade": 88.0},
    {"studentId": 9, "finalGrade": 93.9},
    {"studentId": 10, "finalGrade": 92.5}
  ]
}

First we note the URL of where the data resides. Then we pass that as a parameter to the get function from the requests package, to issue an HTTP GET request:

import requests

request_url = ("https://raw.githubusercontent.com/" +
              "prof-rossetti/intro-software-dev-python-book/main/docs/data/" +
              "gradebook.json")

response = requests.get(request_url)
print(type(response))
<class 'requests.models.Response'>

We receive a response object from the server. The response object has a few methods and properties we care about. First, the status code tells us if the request was generally successful:

response.status_code
200

The text property is a string that looks like it could one day be some data:

print(type(response.text))
response.text
<class 'str'>
'{\n  "downloadDate": "2018-06-05",\n  "professorId": 123,\n  "students": [\n    {"studentId": 1, "finalGrade": 76.7},\n    {"studentId": 2, "finalGrade": 85.1},\n    {"studentId": 3, "finalGrade": 50.3},\n    {"studentId": 4, "finalGrade": 89.8},\n    {"studentId": 5, "finalGrade": 97.4},\n    {"studentId": 6, "finalGrade": 75.5},\n    {"studentId": 7, "finalGrade": 87.2},\n    {"studentId": 8, "finalGrade": 88.0},\n    {"studentId": 9, "finalGrade": 93.9},\n    {"studentId": 10, "finalGrade": 92.5}\n  ]\n}\n'

The final step is to convert this JSON-formatted string to actual Python data. We can do this using the json method of the response object, or by leveraging the loads function from the json module:

data = response.json()
print(type(data))
data
<class 'dict'>
{'downloadDate': '2018-06-05',
 'professorId': 123,
 'students': [{'studentId': 1, 'finalGrade': 76.7},
  {'studentId': 2, 'finalGrade': 85.1},
  {'studentId': 3, 'finalGrade': 50.3},
  {'studentId': 4, 'finalGrade': 89.8},
  {'studentId': 5, 'finalGrade': 97.4},
  {'studentId': 6, 'finalGrade': 75.5},
  {'studentId': 7, 'finalGrade': 87.2},
  {'studentId': 8, 'finalGrade': 88.0},
  {'studentId': 9, 'finalGrade': 93.9},
  {'studentId': 10, 'finalGrade': 92.5}]}
import json

data = json.loads(response.text)
print(type(data))
data
<class 'dict'>
{'downloadDate': '2018-06-05',
 'professorId': 123,
 'students': [{'studentId': 1, 'finalGrade': 76.7},
  {'studentId': 2, 'finalGrade': 85.1},
  {'studentId': 3, 'finalGrade': 50.3},
  {'studentId': 4, 'finalGrade': 89.8},
  {'studentId': 5, 'finalGrade': 97.4},
  {'studentId': 6, 'finalGrade': 75.5},
  {'studentId': 7, 'finalGrade': 87.2},
  {'studentId': 8, 'finalGrade': 88.0},
  {'studentId': 9, 'finalGrade': 93.9},
  {'studentId': 10, 'finalGrade': 92.5}]}

This data happens to be dictionary-like at the top level. However be aware, when we fetch JSON, it can be either list-like, or dictionary-like on the top level. So you must observe the structure of your particular data before processing it further.

students = data["students"]
print(type(students))
len(students)
<class 'list'>
10

Looping through the items:

for student in students:
    print(student["studentId"], student["finalGrade"])
1 76.7
2 85.1
3 50.3
4 89.8
5 97.4
6 75.5
7 87.2
8 88.0
9 93.9
10 92.5

Calculating the average grade:

from statistics import mean, median

grades = [student["finalGrade"] for student in students]

print(mean(grades))
print(median(grades))
83.64
87.6