1
00:00:05,230 --> 00:00:10,870
In the last video, we saw how to use the os.walk generator to work with files and directories

2
00:00:10,870 --> 00:00:12,610
on the local file system.

3
00:00:12,700 --> 00:00:17,680
In this video we're going to use the os.walk in generators of our own, to give a lot of flexibility

4
00:00:17,680 --> 00:00:22,040
in working with files. I'm gonna start out by creating a new file.

5
00:00:22,080 --> 00:00:26,170
So I'm going to do that; new Python file.

6
00:00:26,250 --> 00:00:31,090
I'm going to call this one filesearch.py.

7
00:00:31,950 --> 00:00:38,460
Alright, so let's go ahead and enter the following generator, so start with import os. Then we're going to do import

8
00:00:39,420 --> 00:00:44,720
fnmatch, and we want to start with a function, so
9

9
00:00:44,750 --> 00:00:46,860
Def find underscore albums

10
00:00:49,350 --> 00:00:56,290
parentheses root comma space then artist underscore name, close parentheses and a colon.

11
00:00:56,570 --> 00:01:06,210
Then we're going to put for path comma space directories comma space files in os.walk, root

12
00:01:07,410 --> 00:01:18,390
in parentheses and then a colon. Then we're gonna do for artist in directories colon, then we're gonna put subdir is

13
00:01:18,390 --> 00:01:28,410
equal to os.path.join parentheses path comma space artists, sorry artist, because we've

14
00:01:28,430 --> 00:01:31,170
defined artist on 1ine 7.

15
00:01:31,500 --> 00:01:41,490
And then we're going to do for space album underscore path comma space albums comma space underscore

16
00:01:41,700 --> 00:01:47,550
space in os.walk, subdir in parentheses colon. Then we're going to put

17
00:01:50,380 --> 00:01:59,970
for space album in albums colon. Then we're going to type in yield space

18
00:02:00,080 --> 00:02:03,150
os.path.join.

19
00:02:04,150 --> 00:02:13,990
Then it's going to be album underscore path comma space album, end of parentheses, then comma space album.

20
00:02:15,200 --> 00:02:15,900
OK.

21
00:02:16,320 --> 00:02:19,680
So this generator's similar to the earlier ones. So it uses

22
00:02:19,730 --> 00:02:24,070
os.walk, as you can see there, to walk through the directories. For each one,

23
00:02:24,090 --> 00:02:31,200
it joins a path to the directory name, then walks that tree to pick up the albums. Now at the outer level,

24
00:02:31,230 --> 00:02:34,040
we're not actually interested in the list of files that's returned -

25
00:02:34,170 --> 00:02:34,880
this one here,

26
00:02:35,810 --> 00:02:40,580
line 9 - and that's why I've used the underscore as the name. As I mentioned earlier,

27
00:02:40,660 --> 00:02:45,700
that's a Python convention to show that you're not using a value, but have to specify a value when doing

28
00:02:45,700 --> 00:02:51,040
things like unpacking a tuple or dealing with values returned by a function call.

29
00:02:51,130 --> 00:02:57,760
Now for this example, we return a tuple containing the album name and the path to its songs. Just like the

30
00:02:57,760 --> 00:03:01,800
os.path.split function was used to split a path up,

31
00:03:01,810 --> 00:03:09,730
we use os.path.join to recreate the path to the album directories. So let's add a

32
00:03:09,730 --> 00:03:11,550
simple loop to check that it works.

33
00:03:11,650 --> 00:03:14,930
Then we're going to actually start doing something interesting with it.

34
00:03:15,310 --> 00:03:17,050
So I'm going to type here,

35
00:03:17,250 --> 00:03:22,630
album_list is equal to find_albums,

36
00:03:25,810 --> 00:03:30,960
music in double quotes, and obviously, this is in parentheses and that's the path. And

37
00:03:30,970 --> 00:03:38,360
then the artist we're going to search for is Aerosmith, so capital E, capital A rather, e r o s m i t h, close

38
00:03:38,380 --> 00:03:42,260
parentheses. Then we're gonna type for space

39
00:03:42,280 --> 00:03:48,040
a in album_list colon, print a in parentheses.

40
00:03:48,460 --> 00:03:49,460
So if we run this program now, go back

41
00:03:54,050 --> 00:03:57,830
up to the top. So that's nothing we haven't seen before.

42
00:03:57,830 --> 00:04:03,800
We've got a series of tuples containing the path to the directory, as well as the album name. But where

43
00:04:03,800 --> 00:04:08,630
it gets really interesting is when we start matching file names. So you may have wondered why we got

44
00:04:08,630 --> 00:04:10,490
an unused import at the start of the file, 

45
00:04:10,510 --> 00:04:12,720
and I'll just close this run window down.

46
00:04:12,940 --> 00:04:17,550
If you didn't, you should certainly have wondered why we're passing the name of the band Aerosmith, as an

47
00:04:17,550 --> 00:04:20,980
argument to find_albums, when in fact we're not using it.

48
00:04:21,100 --> 00:04:27,610
So the file name match module, fnmatch, well that's used to match files whose names follow some

49
00:04:27,610 --> 00:04:28,810
sort of pattern.

50
00:04:28,840 --> 00:04:34,750
So our find_albums generator might be more useful if it just returned the albums for a single

51
00:04:34,750 --> 00:04:38,370
artist, and with a quick change we can get it to do just that.

52
00:04:38,370 --> 00:04:42,400
So we can come up here, under the, well actually on line 7.

53
00:04:42,550 --> 00:04:49,980
We've got for artist in directories. We could change that to put for artist in. Then we could type fnmatch dot

54
00:04:50,240 --> 00:04:54,700
filter, then directories will still be in parentheses.

55
00:04:54,960 --> 00:05:02,390
Then we add a comma, then a space, then the artist_name, close parentheses and leave the colon there.

56
00:05:02,770 --> 00:05:07,550
So instead of just looping through all the directories, we can actually filter them first.

57
00:05:07,600 --> 00:05:13,870
So in this case, we'll now get only those albums from directories that exactly match the artist_name that

58
00:05:13,870 --> 00:05:16,000
was passed to this function.

59
00:05:16,570 --> 00:05:22,400
So if we actually run this. Let's check that it works, and obviously, I can't spell because that should have been filter, filter.

60
00:05:22,410 --> 00:05:25,860
That's better.

61
00:05:26,040 --> 00:05:28,670
I wasn't really paying attention to the warning there.

62
00:05:29,450 --> 00:05:33,430
If we run this again now, you can see in this case, we'll just let it run til it finishes,

63
00:05:33,460 --> 00:05:39,040
but at this time, it's not just outputting everything. We've only got some output based on Aerosmith.

64
00:05:39,430 --> 00:05:43,770
So the generator returns a tuple, and you can see the program's exited there.

65
00:05:44,080 --> 00:05:49,810
So the generator retains a tuple containing the path to the directory, and the name of the directory.

66
00:05:50,140 --> 00:05:54,720
So depending on what you want to do with the results, you could just return the album name for argument's

67
00:05:54,720 --> 00:05:55,310
sake.

68
00:05:55,660 --> 00:06:00,990
So in this case, I want to demonstrate how using generators can be a great way to simplify code.

69
00:06:01,030 --> 00:06:02,560
So as an example,

70
00:06:02,560 --> 00:06:06,730
I think it would be more useful to have a way to get the list of songs on the albums.

71
00:06:07,150 --> 00:06:13,330
So let's add the find_songs generator after find_albums, and need to make sure

72
00:06:13,330 --> 00:06:15,000
we leave two blank lines between them as well.

73
00:06:15,010 --> 00:06:18,340
So let's go ahead and do that.

74
00:06:19,390 --> 00:06:27,500
So we're going to type def space find_songs, and the parameter's going to be albums parentheses

75
00:06:27,550 --> 00:06:28,520
colon.

76
00:06:29,230 --> 00:06:38,570
I'm going to type for space album in albums colon, then on the next line for space song space n space os

77
00:06:38,580 --> 00:06:39,490
dot list

78
00:06:39,490 --> 00:06:45,570
dir, parentheses album, then left and right square bracket and a zero.

79
00:06:45,940 --> 00:06:48,840
This is because we want the path, not the name of the album.

80
00:06:49,310 --> 00:06:52,070
So let's put a colon there, and just put a comment there to that effect.

81
00:06:59,760 --> 00:07:02,820
And then after that, we're going to yield song. OK,

82
00:07:06,390 --> 00:07:12,470
and then we do album_list equals find_albums, music Aerosmith as it was before.

83
00:07:12,750 --> 00:07:17,520
But this time we know the directory and we aren't expecting any subdirectories, so there's no need in

84
00:07:17,520 --> 00:07:23,430
that case, to walk the entire directory structure, and we can use the os.listdir instead

85
00:07:23,430 --> 00:07:26,550
of os.walk and that's why I've done that on line 16.

86
00:07:27,000 --> 00:07:32,050
So this should be more efficient but it actually may not be. I'm gonna come back to that point shortly.

87
00:07:32,100 --> 00:07:37,810
But first let's make use of find_songs to get the song list of all Aerosmith albums.

88
00:07:38,100 --> 00:07:40,010
So I'm going to do that on the next line:

89
00:07:40,470 --> 00:07:50,280
song_list is equal to find_songs, album_list. And I'm going to comment out

90
00:07:50,280 --> 00:07:50,290
this loop,

91
00:07:50,290 --> 00:07:52,150
because

92
00:07:52,210 --> 00:08:00,950
we've seen that before, to save a bit of time, and we're going to put for s in song_list colon print,

93
00:08:01,050 --> 00:08:02,380
s in parentheses.

94
00:08:02,700 --> 00:08:09,790
So if we actually run this now, you can see here that we've got all the information we actually need.

95
00:08:09,800 --> 00:08:14,150
The interesting thing here is that we've removed the generator of the data from the code that makes

96
00:08:14,150 --> 00:08:17,890
use of the data. Now we could do that with ordinary functions,

97
00:08:17,980 --> 00:08:24,620
but then we'd be returning lists and they could actually get huge, and they'd also stay in memory until we explicitly

98
00:08:24,620 --> 00:08:27,770
cleared them, which is just something else to forget to do.

99
00:08:28,070 --> 00:08:33,830
So when using this approach, we can create a list of songs for any artist in just four lines of code,

100
00:08:34,220 --> 00:08:36,110
with very little overhead.

101
00:08:36,110 --> 00:08:43,990
So we're basically chaining generators together by passing one generator on, as the argument to the next.

102
00:08:44,000 --> 00:08:45,910
So I'm just closing this again.

103
00:08:45,950 --> 00:08:48,680
So basically, on line 15 here,

104
00:08:49,070 --> 00:08:55,220
when we loop through all those albums, the generator returns the details for each item as we request

105
00:08:55,220 --> 00:08:55,750
them.

106
00:08:55,760 --> 00:09:01,430
So at no time does a complete list of the albums exist in memory, and the same is true of all the songs.

107
00:09:01,890 --> 00:09:02,210
We get

108
00:09:02,210 --> 00:09:07,950
each one as it's needed and the program doesn't create a list of songs as such.

109
00:09:08,390 --> 00:09:15,110
Well I guess that's not strictly true, because the os.walk function does return lists in its tuple. If that's

110
00:09:15,170 --> 00:09:16,580
likely to be a problem,

111
00:09:16,740 --> 00:09:20,670
os.walk may not be suitable for your particular application.

112
00:09:20,700 --> 00:09:25,760
We'll be looking at scandir in a later video, and that can be useful if you want to avoid generating

113
00:09:25,760 --> 00:09:33,520
the lists that os.walk creates. So creating and destroying a lot of lists will affect performance.

114
00:09:33,520 --> 00:09:38,700
So Python has to take care of reclaiming the memory for the previous album song list,

115
00:09:38,701 --> 00:09:38,702
.

116
00:09:38,720 --> 00:09:44,600
each time we process a new album. So with thousands of albums, that memory management is going to have an

117
00:09:44,600 --> 00:09:46,480
effect on performance.

118
00:09:46,490 --> 00:09:51,440
Alright, so using generators like this could be very useful for things like interrogating log files

119
00:09:51,440 --> 00:09:57,910
on a server. So there could be thousands of logs, and patch logs for example, can be really huge.

120
00:09:58,160 --> 00:10:04,210
By using a generator to return the full path and name of each file, then another one to check for a file,

121
00:10:04,210 --> 00:10:04,570
or check

122
00:10:04,580 --> 00:10:11,000
a file for a particular matching string, is a very memory efficient way to look for particular errors, or

123
00:10:11,000 --> 00:10:15,800
requests say, from a certain IP address or whatever you want it to find from the logs.

124
00:10:16,020 --> 00:10:20,750
So generators are really one of those features that make Python an excellent choice for working with

125
00:10:20,750 --> 00:10:22,130
huge data sets.

126
00:10:22,130 --> 00:10:26,430
Once you get the hang of them they're actually quite simple but yet very powerful.

127
00:10:26,500 --> 00:10:30,310
Now there's more you can do with generators since Python 3.5.

128
00:10:30,470 --> 00:10:36,590
It's also possible to set a value into a generator using a slightly different form of yield but that's

129
00:10:36,590 --> 00:10:38,780
enough to be getting on with for now.

130
00:10:38,870 --> 00:10:45,600
So I'm going to finish this video now with a quick comment about the fnmatch module's filter method.

131
00:10:45,620 --> 00:10:51,590
Now I can take wildcards, so it can process all the artists whose names start with black, by adding an

132
00:10:51,590 --> 00:10:52,390
asterix.

133
00:10:52,400 --> 00:10:53,460
Let's go ahead and do that.

134
00:10:53,480 --> 00:11:01,630
For example line 20 here, where we've got Aerosmith, if we change that to capital B L A C K and put an * there,

135
00:11:01,730 --> 00:11:03,920
after that, if we actually run this

136
00:11:06,870 --> 00:11:09,200
you can see we can see different output that we got last time.

137
00:11:09,390 --> 00:11:14,790
We get all the songs for the Black Crowes black case black hole Arkansas or Black River Project.

138
00:11:14,790 --> 00:11:17,690
Black Sabbath and black beards Tea-Party.

139
00:11:17,910 --> 00:11:21,720
So that's pretty cool and makes it easy to find all .log files

140
00:11:21,730 --> 00:11:23,310
In the example I mentioned just now.

141
00:11:23,460 --> 00:11:24,910
Searching log files.

142
00:11:25,170 --> 00:11:26,870
There is one thing to be aware of though.

143
00:11:27,030 --> 00:11:29,900
The future is sort of case sensitive.

144
00:11:30,000 --> 00:11:36,270
Now that sounds a little bit vague its because the behavior depends on the underlying file system.

145
00:11:36,270 --> 00:11:41,400
Now Linux file system is a case sensitive so it's not surprising that if it matches all saké sensitive

146
00:11:41,400 --> 00:11:45,300
on Linux but Windows NTFS isn't case sensitive.

147
00:11:45,300 --> 00:11:48,320
So if I change black to lower case.

148
00:11:48,390 --> 00:11:50,690
In other words it closes down.

149
00:11:50,980 --> 00:11:53,590
Change it to a lower case pay for black.

150
00:11:53,640 --> 00:11:59,200
The program will still return the same results on Windows as long as I'm using an NTFS drive of course

151
00:11:59,200 --> 00:12:02,830
so that is a hard drive that's been formatted with NTFS.

152
00:12:02,890 --> 00:12:06,810
Now if you connect remotely to a Linux drive then the behaviour would be different.

153
00:12:07,230 --> 00:12:12,340
So I actually run this, in my case because I'm on a Mac.

154
00:12:12,530 --> 00:12:13,930
This results in no output.

155
00:12:14,150 --> 00:12:18,520
But that example examples a change with the lower case b would still work in Windows.

156
00:12:18,530 --> 00:12:23,270
Now I haven't mentioned the Mac yet and I should point out that I haven't changed the behavior of my

157
00:12:23,270 --> 00:12:24,530
next file system.

158
00:12:24,890 --> 00:12:30,440
So you might expect that the program to work for me is well after all the Mac file system HFS isn't

159
00:12:30,440 --> 00:12:33,230
case sensitive by default.

160
00:12:33,260 --> 00:12:35,620
It is however case preserving.

161
00:12:35,660 --> 00:12:40,000
So there's a subtle difference between case sensitive and case preserving.

162
00:12:40,010 --> 00:12:44,180
So if you wrote a Python program that will run on different operating systems you need to be really

163
00:12:44,180 --> 00:12:45,450
aware of that.

164
00:12:45,470 --> 00:12:47,260
So with HFS or HFS

165
00:12:47,270 --> 00:12:54,050
Plus, you can't create files with the same name using different case but you can refer to a file

166
00:12:54,050 --> 00:12:56,900
using its name and either uppercase or lowercase.

167
00:12:57,110 --> 00:13:02,350
When Python reads the file it returns the name in the original case that it was created with.

168
00:13:02,390 --> 00:13:05,670
So that's why black in lowercase isn't matching our directories.

169
00:13:05,900 --> 00:13:08,660
Even though HFS isn't case sensitive.

170
00:13:08,660 --> 00:13:14,420
And again HSF and HSF+ are file systems on a Mac. Now you shouldn't need to do this for the

171
00:13:14,420 --> 00:13:15,190
next challenge.

172
00:13:15,200 --> 00:13:20,960
But if you want to cater for matching black regardless of case, then what you could do is use a generator

173
00:13:20,960 --> 00:13:27,970
expression to convert the directory names to uppercase. Generator expressions will make much more sense

174
00:13:27,980 --> 00:13:30,040
when we've covered list comprehensions.

175
00:13:30,040 --> 00:13:32,570
So really don't worry too much about his next bit of code.

176
00:13:32,660 --> 00:13:38,500
I'm just including it here to show you one way of dealing with this, rather than leaving you hanging. Please

177
00:13:38,500 --> 00:13:41,260
don't ask questions about this generator expression at this point.

178
00:13:41,370 --> 00:13:45,110
And that's because it's going to be explained in detail in a later video.

179
00:13:45,520 --> 00:13:47,480
So let's actually have a look at that.

180
00:13:47,490 --> 00:13:50,390
So what I'm going to do first is comment out of the existing.

181
00:13:50,560 --> 00:13:56,390
If you matched up filter. So for what you that what I would do is actually duplicate that line and coming up

182
00:13:56,390 --> 00:13:59,990
in the first one you want to make a change it is going to be for artists.

183
00:14:00,040 --> 00:14:01,850
If you matched futa.

184
00:14:02,050 --> 00:14:05,510
But here is where it changes and I'm just going to delete that and just top it again.

185
00:14:05,860 --> 00:14:09,010
So I went to see teleprinter CS D.

186
00:14:09,080 --> 00:14:16,550
Oppa then left to rot parentheses for d in.

187
00:14:17,950 --> 00:14:25,930
Single parentheses Raupp prophecies that is Colma spice capsule's score 9 and another wrong parentheses

188
00:14:25,930 --> 00:14:27,050
and then a colon.

189
00:14:27,330 --> 00:14:31,850
I am actually adding another line here after the method declaration.

190
00:14:31,960 --> 00:14:37,830
The function declaration at the top the caps is called 9 is equal to artist.

191
00:14:38,090 --> 00:14:46,680
Don't name drop us and then let the parentheses to be honest on the score top line.

192
00:14:47,700 --> 00:14:48,550
If I run this

193
00:14:51,420 --> 00:14:56,040
you can say where are getting a working again on my map but tonight this still doesn't work on case

194
00:14:56,040 --> 00:15:01,890
sensitive file systems such as Linux uses the case sensitive file systems it's probably best to use

195
00:15:01,890 --> 00:15:03,700
regular expressions.

196
00:15:03,750 --> 00:15:08,910
However a slightly more complicated generator expression will actually do the job but isn't a sufficient

197
00:15:08,970 --> 00:15:10,610
as using future.

198
00:15:10,720 --> 00:15:14,370
So let's have a look at that so I'm going to close this down and I'm going to come into that second

199
00:15:14,370 --> 00:15:15,400
line again.

200
00:15:15,990 --> 00:15:22,050
Let's talk an example which is going to be for us in in parentheses.

201
00:15:22,050 --> 00:15:26,560
T for t in directories.

202
00:15:27,430 --> 00:15:39,030
If it didn't match dot if it match in parentheses data Anapa comma caps underscore 9 and to what parentheses

203
00:15:39,030 --> 00:15:40,120
and then a colon.

204
00:15:40,460 --> 00:15:43,230
So that's more complex but that should actually work.

205
00:15:43,230 --> 00:15:44,920
So just checked and it still works on a Mac.

206
00:15:45,240 --> 00:15:49,290
And you find this will also work on Linux and obviously it's going to work on Windows anyway because

207
00:15:49,290 --> 00:15:55,120
it's a case insensitive so they generate expressions in case you're wondering will make a lot more sense

208
00:15:55,130 --> 00:15:56,850
a bit later in the course.

209
00:15:56,870 --> 00:16:01,390
Right so let's end the video here in the next video I'm going to start you off with a challenge.

210
00:16:01,460 --> 00:16:06,290
Then look at another way for extracting the information we want without relying on having such a neatly

211
00:16:06,290 --> 00:16:08,060
organized directory structure.

212
00:16:08,150 --> 00:16:12,100
After all who has a music collection is organized I feel they had drives.

213
00:16:12,110 --> 00:16:13,300
See you in the next video.