1
00:00:05,120 --> 00:00:09,200
In the next 2 videos, we're going to work 
with some real data from the internet. 

2
00:00:09,200 --> 00:00:13,520
I'll start by reading the data from a file on 
the local disk drive, then in the next video, 

3
00:00:13,520 --> 00:00:17,280
we'll download and parse the data.
The data we'll use comes from a US 

4
00:00:17,280 --> 00:00:21,520
government organisation, the National 
Oceanic and Atmospheric Administration., 

5
00:00:21,520 --> 00:00:25,460
and we'll process some data they've 
collected on variances in global temperature. 

6
00:00:25,520 --> 00:00:29,840
I'll point my browser at the website 
where they've published the data. 

7
00:00:30,720 --> 00:00:34,000
Just in case this site becomes 
unavailable, I've put the data into 

8
00:00:34,000 --> 00:00:38,623
the file temperature_anomaly.json, 
in the resources for this video. 

9
00:00:38,720 --> 00:00:43,760
Download that file, and place it into 
your project directory, for this example. 

10
00:00:43,760 --> 00:00:49,200
If you want to generate the data on this 
page, change the End Year to 2021 – or later, 

11
00:00:49,200 --> 00:00:54,960
if you're watching this video after 2021.
After making any changes, you have to click 

12
00:00:54,960 --> 00:01:00,320
the Plot button, to update the page. The 
only change I've made is to the End Year. 

13
00:01:00,320 --> 00:01:05,122
Scroll down to the table after the graph, and 
there are download links just before the table. 

14
00:01:05,200 --> 00:01:10,887
There are options to download as 
XML, CSV, and – the last one – JSON. 

15
00:01:10,960 --> 00:01:15,040
Most browsers will show the URL for links, 
usually at the bottom of the browser window, 

16
00:01:15,040 --> 00:01:19,680
when you hover over a link on a web page.
Hover over the JSON link, and you'll see 

17
00:01:19,680 --> 00:01:25,120
the URL that we'll use, in the next video. If 
you need to copy a link like this, to work with 

18
00:01:25,120 --> 00:01:29,920
some different data, you can right-click and 
choose the option to copy the link address. 

19
00:01:29,920 --> 00:01:34,064
Ok, that's the source of our 
data, now let's write the code. 

20
00:01:34,240 --> 00:01:39,440
Back in IntelliJ, I've created a new 
Python file called global_temps.py 

21
00:02:16,000 --> 00:02:21,171
We have to import the json module, 
then we open the file in text mode, for reading. 

22
00:02:21,171 --> 00:02:25,358
Those are the default modes, 
so I haven't specified them explicitly. 

23
00:02:25,358 --> 00:02:28,640
I have specified the encoding though. As we saw, 

24
00:02:28,640 --> 00:02:33,360
in the video on File Encodings, it's a good 
idea to be explicit about the encoding. 

25
00:02:33,360 --> 00:02:40,000
On line 6, we use the json module's load command, 
to read the data from the file, and decode it. 

26
00:02:40,000 --> 00:02:42,560
Now we have to decide what 
we're going to do with it. 

27
00:02:42,560 --> 00:02:46,800
Because JSON doesn't describe the data 
it contains – it just contains data, 

28
00:02:46,800 --> 00:02:52,093
with no indication of what it means – you'll 
almost always have to check the data yourself. 

29
00:02:52,480 --> 00:02:57,749
Open temperature_anomaly.json in IntelliJ, 
and let's see what we're working with. 

30
00:02:57,840 --> 00:03:02,449
We can see that it starts with a dictionary 
called description, at the start of line 1. 

31
00:03:02,560 --> 00:03:06,720
But this data isn't very easy to 
read, because it's all on one line. 

32
00:03:06,720 --> 00:03:10,539
Fortunately, modern editors can 
often help with things like this. 

33
00:03:10,640 --> 00:03:16,035
Even more fortunately, as I mentioned earlier, 
JSON doesn't care about extra whitespace. 

34
00:03:16,240 --> 00:03:20,722
So reformatting this data isn't going to 
cause us any problems with de-serializing it. 

35
00:03:20,880 --> 00:03:26,080
From the Code menu, choose Reformat Code.
Check the shortcut, when you go into these 

36
00:03:26,080 --> 00:03:30,949
menu options, and you'll probably remember 
the shortcut for options that you use a lot. 

37
00:03:30,949 --> 00:03:36,400
Ok, that's a lot better to work with.
At the top of the document is the description key, 

38
00:03:36,400 --> 00:03:40,240
whose value is a dictionary. 
That briefly describes the data, 

39
00:03:40,240 --> 00:03:45,440
which is useful, and often doesn't happen.
I'm guessing now, but it looks like missing data 

40
00:03:45,440 --> 00:03:51,840
will be represented by the value -999. As there 
doesn't seem to be any missing data, we don't 

41
00:03:51,840 --> 00:03:56,727
have to worry about that, but it would be a useful bit 
of information if there were any missing values. 

42
00:03:56,800 --> 00:04:02,240
Below that, the actual data values are in another 
dictionary, which is the value for the key data. 

43
00:04:02,240 --> 00:04:08,080
That shows the annual average temperature 
anomalies for each year, from 1880 to 2020. 

44
00:04:08,080 --> 00:04:13,360
You might have additional years, of course, if 
you generated and downloaded your data later. 

45
00:04:13,360 --> 00:04:16,839
Note that the years and temperature 
anomalies are stored as strings. 

46
00:04:16,839 --> 00:04:22,720
Ideally, we'd want integers and floats, but converting 
these strings in our Python code is easy. 

47
00:04:22,720 --> 00:04:27,920
Right at the end of the file, we've added the 
citation – where the data came from. That's only 

48
00:04:27,920 --> 00:04:32,703
polite, and might even be a legal requirement, 
depending on where you get your data from. 

49
00:04:32,800 --> 00:04:36,960
That citation won't be present when we download 
the data, it's something that we manually added 

50
00:04:36,960 --> 00:04:42,000
to the file, after downloading the data.
As I said, if you're going to re-distribute 

51
00:04:42,000 --> 00:04:45,921
someone else's data, and assuming 
they've given you permission to do so, 

52
00:04:45,921 --> 00:04:49,019
then it's polite to let people 
know where the data came from. 

53
00:04:49,280 --> 00:04:53,440
Alright, we now know what we're working with.
When we de-serialize this data, 

54
00:04:53,440 --> 00:04:58,480
we should get a dictionary with three 
keys: description, data and citation. 

55
00:04:58,480 --> 00:05:01,974
The values for those first two 
keys are themselves dictionaries. 

56
00:05:02,080 --> 00:05:07,247
We can now start to process the data.
Switch back to global_temps.py, 

57
00:05:07,247 --> 00:05:09,726
and I'll start by printing the description:

58
00:05:32,915 --> 00:05:38,274
When working with strange data like this, it can be
helpful to access and print something from the data. 

59
00:05:38,274 --> 00:05:43,600
That will give you some confidence that it's been parsed
correctly, and that you are working with what you expect. 

60
00:05:43,600 --> 00:05:49,360
Let's run the program, and see what we get.
That looks good! We've got a dictionary containing 

61
00:05:49,360 --> 00:05:54,422
the four keys and values that we saw in the data.
Below that is the citation. 

62
00:05:54,422 --> 00:05:59,360
So it looks like we've correctly identified what's in 
the data, and it's parsed successfully. 

63
00:05:59,360 --> 00:06:03,320
We can now iterate over the data, 
and print the values for each year: 

64
00:06:53,360 --> 00:06:58,640
Line 11 converts the string values to integer 
and float, for year and value respectively. 

65
00:06:58,640 --> 00:07:04,000
I've used some formatting options, on line 12, to 
print the values in a field width of 6 characters, 

66
00:07:04,000 --> 00:07:07,891
aligned to the right.
Run the program 

67
00:07:08,800 --> 00:07:12,400
and scroll up through the data, 
to check that it looks ok. 

68
00:07:12,400 --> 00:07:18,880
We've got values for all years, from 1880 
to 2020, neatly presented in columns. 

69
00:07:18,880 --> 00:07:23,520
Ok, we've read data from a file, 
and successfully de-serialized it. 

70
00:07:23,520 --> 00:07:27,680
We've now got the data in a form 
that we can use, in our Python code. 

71
00:07:27,680 --> 00:07:31,988
Our data was stored in a local file, 
but that means it will go out of date. 

72
00:07:32,160 --> 00:07:36,160
Often, you'll want to make sure you're 
working with the latest data available. 

73
00:07:36,160 --> 00:07:39,280
In that case, you'd want to 
download the JSON from the internet, 

74
00:07:39,280 --> 00:07:44,233
rather than storing it locally.
We'll see how to do that, in the next video.