1
00:00:05,120 --> 00:00:08,505
In this video, we'll do something 
similar to the last one. 

2
00:00:08,505 --> 00:00:12,125
We're going to take the JSON data for the 
temperature anomalies over the years, 

3
00:00:12,125 --> 00:00:17,414
and put it into a format that we can work with in Python.
That's what we did in the previous video. 

4
00:00:17,520 --> 00:00:22,540
The difference, this time, is that we'll 
download the JSON directly from the internet. 

5
00:00:23,840 --> 00:00:28,880
In today's connected world, data no longer 
has to exist on your local file system. 

6
00:00:28,880 --> 00:00:32,708
It could come from another computer 
on your network, or from the internet. 

7
00:00:32,708 --> 00:00:37,348
It's increasingly common for data to be 
stored on remote servers, or in the "cloud". 

8
00:00:37,440 --> 00:00:41,728
This doesn't mean we have to do 
things completely differently though. 

9
00:00:43,280 --> 00:00:48,000
A long long time ago, when the world was young ...
The designers of languages like C, 

10
00:00:48,000 --> 00:00:52,083
which is one of the languages that 
inspired Python, worked with streams. 

11
00:00:52,160 --> 00:00:55,931
When reading from a text file, 
we actually use a text stream. 

12
00:00:56,000 --> 00:01:01,280
It gets cleverer than that – when getting data 
from the keyboard, C uses an input stream. 

13
00:01:01,280 --> 00:01:06,480
That makes it trivial to change a program, 
so that input can come from a file, instead. 

14
00:01:06,480 --> 00:01:11,680
It's also trivial to send output to a file, 
instead of to the console. And we saw an example 

15
00:01:11,680 --> 00:01:17,455
of that, when we included a file argument to the 
print function, so that we could write to a file. 

16
00:01:19,520 --> 00:01:24,720
That design decision was made long before the 
internet existed, and is extremely flexible. 

17
00:01:24,720 --> 00:01:27,440
We read from a stream that's 
connected to an input source, 

18
00:01:27,440 --> 00:01:30,948
and write to a stream that's 
connected to an output destination. 

19
00:01:31,040 --> 00:01:34,160
By associating our text streams 
to a different input or output, 

20
00:01:34,160 --> 00:01:40,880
we can get data from (or send data to) different 
places, without making huge changes to our code. 

21
00:01:40,880 --> 00:01:46,022
In this example, we'll open a URL, and 
get our JSON data from the internet. 

22
00:01:48,080 --> 00:01:51,360
When we were reading and writing 
to files, in the earlier examples, 

23
00:01:51,360 --> 00:01:55,529
I didn't include any error handling.
We'll do the same in this example: 

24
00:01:55,529 --> 00:02:01,440
we're not going to deal with any errors that could happen.
The purpose of these examples is to focus on I/O. 

25
00:02:01,440 --> 00:02:05,520
We'll cover error handling, in 
detail, later in the course. 

26
00:02:05,520 --> 00:02:08,400
I don't want to overcomplicate 
things at this stage. 

27
00:02:08,400 --> 00:02:12,930
So please be aware that production 
quality code would include error handling. 

28
00:02:13,840 --> 00:02:19,600
Ok, back in IntelliJ, we'll make some 
changes to our global_temps Python code. 

29
00:02:19,600 --> 00:02:23,200
I'm going to use the urllib.request 
module for this example. 

30
00:02:23,200 --> 00:02:28,320
Python includes a basic library for opening 
URLs. When you check out the documentation 

31
00:02:28,320 --> 00:02:33,945
for this module, you'll see a recommendation to use 
a third-party library called requests, instead. 

32
00:02:34,080 --> 00:02:39,238
Because I won't be using urllib in our real 
examples, this is a good opportunity to show it to you. 

33
00:02:39,238 --> 00:02:43,537
It also means we can write this example 
without having to install another package. 

34
00:02:43,600 --> 00:02:46,930
If you come across Python code 
that uses the urllib library, 

35
00:02:46,930 --> 00:02:52,250
you'll at least be familiar with it.
We start by importing the request module from urllib: 

36
00:02:56,256 --> 00:02:58,400
Next, we replace our file name with 

37
00:02:58,400 --> 00:03:02,720
the URL of the website that hosts the data:
You can get the URL by right-clicking the 

38
00:03:02,720 --> 00:03:06,225
JSON download link, in the web page 
we looked at earlier. 

39
00:03:06,225 --> 00:03:12,264
Alternatively, copy it from the temperature_url.txt 
file in the resources for this video. 

40
00:03:12,480 --> 00:03:17,644
Ok, I'll replace line 6 with the code to 
download the JSON data from the internet: 

41
00:03:29,600 --> 00:03:32,450
The urlopen function can be 
used in a with statement, 

42
00:03:32,450 --> 00:03:35,040
so our internet connection will 
be automatically closed for us. 

43
00:03:35,840 --> 00:03:40,480
The next step is to read the data from the stream 
that we've opened, and decode it from UTF-8: 

44
00:03:53,920 --> 00:03:58,880
Our original line 6, which is now commented out, 
specified the encoding when we opened the file. 

45
00:03:59,680 --> 00:04:04,854
The urlopen function doesn't let us specify 
an encoding, so we decode the data after reading –

46
00:04:04,854 --> 00:04:07,920
that's on line 8.
Other than that minor change, 

47
00:04:07,920 --> 00:04:12,640
there isn't a great deal of difference between 
reading from a file, or reading from a URL. 

48
00:04:12,640 --> 00:04:17,680
There is one other change we need to make.
On line 9, the JSON load function takes care 

49
00:04:17,680 --> 00:04:21,440
of reading the data from a file.
We've read the data on line 8, 

50
00:04:21,440 --> 00:04:27,667
and now have a string containing the JSON.
json.load gets data from a file – as we've seen.

51
00:04:27,667 --> 00:04:34,411
Similarly, json.dump sends data to a file.
There are 2 other functions, corresponding to these, 

52
00:04:34,411 --> 00:04:40,080
that take or return a string. The names 
are almost the same, they just end in "s". 

53
00:04:40,080 --> 00:04:43,593
We'll change line 9 to use 
the JSON loads function: 

54
00:04:43,840 --> 00:04:48,560
Just to be clear, we've now used two 
steps to perform the equivalent of line 6. 

55
00:04:48,560 --> 00:04:54,880
Line 6 passed an encoding to the open function, 
and open decoded the input stream as UTF-8 for us. 

56
00:04:54,880 --> 00:04:58,560
That's a convenience that open provides, 
but we could have read the data, 

57
00:04:58,560 --> 00:05:02,720
and decoded it separately.
That's what we're doing on line 8. 

58
00:05:02,720 --> 00:05:07,280
The urllib library is quite old, and 
doesn't automatically decode for us. 

59
00:05:07,280 --> 00:05:10,809
So we have to explicitly 
decode the data from UTF-8. 

60
00:05:10,880 --> 00:05:14,800
As I've already mentioned, we should 
include quite a lot of error handling. 

61
00:05:14,800 --> 00:05:19,040
There are a lot of things that can go wrong, 
when getting information from the internet. 

62
00:05:19,040 --> 00:05:22,160
The internet connection may have gone 
down, or the remote server that we're 

63
00:05:22,160 --> 00:05:26,240
connecting to might be offline, or 
experiencing some other problem. 

64
00:05:26,240 --> 00:05:30,880
So once again, don't treat this as 
production quality code for downloading data. 

65
00:05:30,880 --> 00:05:35,695
This is just an example of reading data 
from a URL, rather than from a file. 

66
00:05:35,840 --> 00:05:43,680
Ok, let's run the program, and see if it works.
That's largely OK. I have got an error, 

67
00:05:43,680 --> 00:05:47,355
but – scrolling up through the 
output – the data looks pretty good. 

68
00:05:47,355 --> 00:05:52,080
The reason we got an error, is because 
we'd included a citation in our text file. 

69
00:05:52,080 --> 00:05:56,640
As I said, you should be polite, 
and attribute any data that you use. 

70
00:05:56,640 --> 00:06:00,960
That citation doesn't exist in the data 
we've downloaded, so we get an error. 

71
00:06:00,960 --> 00:06:05,840
If we delete line 18:
and run the program again: 

72
00:06:08,640 --> 00:06:14,800
it runs without error.
We've covered quite a lot in the last 6 videos. 

73
00:06:14,800 --> 00:06:20,000
We've seen how using the wrong encoding can 
lead to problems, when reading and writing text data. 

74
00:06:20,000 --> 00:06:24,400
Always try to be explicit about 
the encoding, when you read and write text. 

75
00:06:24,400 --> 00:06:30,560
The UTF-8 encoding, specified by the Unicode 
standard, is a good one to use whenever possible. 

76
00:06:30,560 --> 00:06:35,774
It's the encoding used on the internet, 
and is also the encoding used by JSON. 

77
00:06:37,360 --> 00:06:40,560
UTF-8 doesn't solve all problems, however. 

78
00:06:40,560 --> 00:06:44,480
It does have limitations with Chinese, 
Japanese and Korean alphabets, 

79
00:06:44,480 --> 00:06:49,440
and you may need to use a different encoding 
sometimes, when working in those languages. 

80
00:06:49,440 --> 00:06:54,160
When working with JSON data, you'll often 
need to investigate the data in a text editor, 

81
00:06:54,160 --> 00:06:59,280
to understand what it contains. It's easy 
to check the JSON contents in a text editor, 

82
00:06:59,280 --> 00:07:04,157
which is one reason why it's a popular 
format for exchanging information. 

83
00:07:05,840 --> 00:07:10,400
We introduced the concept of a stream.
You may be familiar with streaming video 

84
00:07:10,400 --> 00:07:15,760
(for example, on YouTube) or streaming 
audio (from Spotify, perhaps). 

85
00:07:15,760 --> 00:07:19,441
In programming terms, a stream can 
be attached to an open file, 

86
00:07:19,441 --> 00:07:24,821
or to a URL on the internet (amongst other things).
Rather than reading from the source directly, 

87
00:07:24,821 --> 00:07:28,800
we use a stream in our Python code.
That allows us to process text in 

88
00:07:28,800 --> 00:07:33,105
a similar manner, regardless of 
the actual source of that text. 

89
00:07:34,880 --> 00:07:36,800
It's useful to be aware of streams, 

90
00:07:36,800 --> 00:07:40,720
because they explain the mechanism that lets 
you easily switch from reading a text file, 

91
00:07:40,720 --> 00:07:45,360
to reading data from an internet URL.
In practice, we generally call a read 

92
00:07:45,360 --> 00:07:51,700
method of the stream, and get back a string – 
or a bytearray, which we'll talk about later. 

93
00:07:52,320 --> 00:07:56,560
Ok, that was a couple of practical 
examples of reading JSON data. 

94
00:07:56,560 --> 00:08:01,195
In the next video, we'll have a look 
at another common file format, CSV. 

95
00:08:01,280 --> 00:08:03,900
I'll see you, in the next video.