WEBVTT 1 00:00:00.270 --> 00:00:01.976 Hey guys welcome back 2 00:00:01.976 --> 00:00:05.728 and in this video we're talking about generators in Python. 3 00:00:05.728 --> 00:00:07.126 I'm really excited for this section 4 00:00:07.126 --> 00:00:09.975 because we're going to talk about some more advanced stuff 5 00:00:09.975 --> 00:00:12.713 and generators are one of these topics in Python that 6 00:00:12.713 --> 00:00:14.675 a lot of courses don't even mention 7 00:00:14.675 --> 00:00:18.535 but they are essential for performance 8 00:00:18.535 --> 00:00:21.927 and also because the e-sync development in Python, 9 00:00:21.927 --> 00:00:22.959 the e-synchronous development 10 00:00:22.959 --> 00:00:24.121 that we're going to be learning later on 11 00:00:24.121 --> 00:00:27.770 is built upon these generators. 12 00:00:27.770 --> 00:00:29.215 When you understand generators 13 00:00:29.215 --> 00:00:32.747 a lot of things in Python start becoming a lot clearer. 14 00:00:32.747 --> 00:00:34.943 So what is a generator? 15 00:00:34.943 --> 00:00:37.922 A generator in Python is a function. 16 00:00:37.922 --> 00:00:39.259 It is a special function. 17 00:00:39.259 --> 00:00:41.966 See a function that remembers the state it's in 18 00:00:41.966 --> 00:00:43.547 in between executions, 19 00:00:43.547 --> 00:00:46.019 so you can run the function multiple times 20 00:00:46.019 --> 00:00:48.197 and it will remember what it did the 21 00:00:48.197 --> 00:00:50.753 last time that you ran it. 22 00:00:50.753 --> 00:00:54.053 So that sounds pretty weird, let me explain with an example. 23 00:00:54.053 --> 00:00:56.493 Imagine you wanted to build a list of 100 numbers, 24 00:00:56.493 --> 00:00:57.967 very straightforward, 25 00:00:57.967 --> 00:01:02.285 you want to get a new list with numbers from zero to 99, 26 00:01:02.285 --> 00:01:04.645 not including 100. 27 00:01:04.645 --> 00:01:06.775 So you could make a function like this one. 28 00:01:06.775 --> 00:01:10.671 Obviously not using range and so forth, you know range. 29 00:01:10.671 --> 00:01:14.863 Let's not use range instead let's build a list from scratch. 30 00:01:14.863 --> 00:01:16.771 How would you do it. 31 00:01:16.771 --> 00:01:20.706 A good way to do it would be start with an empty list. 32 00:01:20.706 --> 00:01:24.751 Let's say call it nums and make it equal to an empty list. 33 00:01:24.751 --> 00:01:28.324 And also start with an index equal to zero. 34 00:01:28.324 --> 00:01:31.441 Then while i is less than 100, 35 00:01:31.441 --> 00:01:34.858 say nums dot append i. 36 00:01:35.768 --> 00:01:38.951 So you're gonna append zero at the start. 37 00:01:38.951 --> 00:01:41.945 Then you're going to increment i by one, 38 00:01:41.945 --> 00:01:43.586 which is gonna make it into one. 39 00:01:43.586 --> 00:01:46.288 You're gonna append one, repeat, append two, 40 00:01:46.288 --> 00:01:50.646 then three and so forth, until you get to 99. 41 00:01:50.646 --> 00:01:52.597 And at the end of this function return nums, 42 00:01:52.597 --> 00:01:55.448 which is your list of numbers. 43 00:01:55.448 --> 00:02:00.448 If we do print hundred numbers and we run that. 44 00:02:01.838 --> 00:02:03.519 We can see that it contains here 45 00:02:03.519 --> 00:02:06.769 our list of a hundred numbers. 46 00:02:07.824 --> 00:02:10.167 Of course we could use list comprehension for this. 47 00:02:10.167 --> 00:02:11.551 We could use the range function, 48 00:02:11.551 --> 00:02:13.240 but for now let's assume this is a cool way of doing it, 49 00:02:13.240 --> 00:02:15.761 because what I'm really explaining is 50 00:02:15.761 --> 00:02:19.710 essentially how the range function works on the inside. 51 00:02:19.710 --> 00:02:20.932 So we construct a list, 52 00:02:20.932 --> 00:02:23.745 fill it with the first hundred numbers and then return. 53 00:02:23.745 --> 00:02:26.846 Now we have a hundred numbers in a list. 54 00:02:26.846 --> 00:02:31.846 This entire list is in your computers memory stored as 55 00:02:32.587 --> 00:02:35.420 this return value of the function. 56 00:02:36.261 --> 00:02:38.804 Now hundred numbers is not very big 57 00:02:38.804 --> 00:02:41.040 it's quite a small amount of space, 58 00:02:41.040 --> 00:02:45.450 but let's say we wanted to store 10 million, 59 00:02:45.450 --> 00:02:48.189 million is like a hundred numbers, 60 00:02:48.189 --> 00:02:52.272 but it would be something like a million numbers. 61 00:02:53.296 --> 00:02:56.039 You would generate a list of a million numbers 62 00:02:56.039 --> 00:02:57.546 and store it in this variable 63 00:02:57.546 --> 00:02:59.936 that goes into your computers memory 64 00:02:59.936 --> 00:03:02.762 and it's presumably a bigger amount of memory 65 00:03:02.762 --> 00:03:05.003 that it's taking, because each one of these numbers 66 00:03:05.003 --> 00:03:08.873 is taking up a small amount of your computers memory. 67 00:03:08.873 --> 00:03:13.337 If you wanted 10 million numbers then it is bigger and so. 68 00:03:13.337 --> 00:03:16.845 Now having 10 million numbers not a great example, 69 00:03:16.845 --> 00:03:20.146 but just imagine you have a very long list of something. 70 00:03:20.146 --> 00:03:22.000 For example, these are often used in 71 00:03:22.000 --> 00:03:24.237 calculating prime numbers and in big data 72 00:03:24.237 --> 00:03:25.849 and things like that you maybe sometimes 73 00:03:25.849 --> 00:03:28.695 want big lists of things. 74 00:03:28.695 --> 00:03:31.853 So a generator is used to circumvent this problem 75 00:03:31.853 --> 00:03:34.979 because the assumption is you don't need 100 numbers 76 00:03:34.979 --> 00:03:36.868 all at once. 77 00:03:36.868 --> 00:03:41.219 What you need is zero so that you can do something with it. 78 00:03:41.219 --> 00:03:42.933 And then one so that you can do something with it. 79 00:03:42.933 --> 00:03:45.855 And then two so that you can do something with it. 80 00:03:45.855 --> 00:03:48.185 For example with this hundred numbers we could then go 81 00:03:48.185 --> 00:03:52.846 and multiply them all by two and store them in another list. 82 00:03:52.846 --> 00:03:54.014 How would you do that? 83 00:03:54.014 --> 00:03:57.097 You'd do something like less comprehension, 84 00:03:57.097 --> 00:04:00.430 x times two for x in hundred numbers. 85 00:04:02.186 --> 00:04:04.119 What you're doing here is you're using this list 86 00:04:04.119 --> 00:04:07.472 to isolate over it, and multiply each number by two. 87 00:04:07.472 --> 00:04:09.396 Let's think that this multiplication by two 88 00:04:09.396 --> 00:04:11.504 is an operation that we want to do in these numbers 89 00:04:11.504 --> 00:04:14.217 in order to calculate something. 90 00:04:14.217 --> 00:04:16.218 Another example would be, if you have a lot of users 91 00:04:16.218 --> 00:04:19.096 and you wanted to calculate the engagement around them, 92 00:04:19.096 --> 00:04:21.026 or how engaged they are with your system, 93 00:04:21.026 --> 00:04:22.924 or something like that. 94 00:04:22.924 --> 00:04:24.774 Essentially you want to go over each element 95 00:04:24.774 --> 00:04:26.857 and perform an operation. 96 00:04:27.709 --> 00:04:31.214 The thing here, is you don't need all the elements at once. 97 00:04:31.214 --> 00:04:33.730 You need them one by one. 98 00:04:33.730 --> 00:04:36.076 So this is what generators are used for. 99 00:04:36.076 --> 00:04:38.718 Instead of having the entire list of elements, 100 00:04:38.718 --> 00:04:40.692 what you do, is the generator gives you 101 00:04:40.692 --> 00:04:42.481 the first element of the list, 102 00:04:42.481 --> 00:04:45.711 without storing all of the list in memory. 103 00:04:45.711 --> 00:04:47.120 The next time you call the generator, 104 00:04:47.120 --> 00:04:49.639 it remembers the element it gave you last, 105 00:04:49.639 --> 00:04:52.154 and it knows to give you the second element. 106 00:04:52.154 --> 00:04:54.624 Then you run it again and it gives you the third element, 107 00:04:54.624 --> 00:04:55.712 and so forth. 108 00:04:55.712 --> 00:04:58.433 It never stores the list in memory, 109 00:04:58.433 --> 00:05:02.664 it always only remembers the last number that it gave you, 110 00:05:02.664 --> 00:05:05.925 so it can then give you the next one. 111 00:05:05.925 --> 00:05:07.522 You have to fun the function every time you want 112 00:05:07.522 --> 00:05:10.225 a new number, and that's why it's called a generator, 113 00:05:10.225 --> 00:05:12.899 because it generates numbers, or indeed strings, 114 00:05:12.899 --> 00:05:17.604 or users or whatever it is that you want to generate. 115 00:05:17.604 --> 00:05:22.604 Here's how you would do it, instead of "nums.append(i)", 116 00:05:22.844 --> 00:05:27.532 you say "yield i", 117 00:05:27.532 --> 00:05:30.075 for i is the number we want to return, 118 00:05:30.075 --> 00:05:33.157 we no longer need this nums equal square bracket, 119 00:05:33.157 --> 00:05:37.157 and we no longer need the return segment either. 120 00:05:38.012 --> 00:05:42.095 So now this function here is called a generator function. 121 00:05:42.095 --> 00:05:46.423 Yield is very much like return, but what happens is 122 00:05:46.423 --> 00:05:51.024 it gives you i when you call it, 123 00:05:51.024 --> 00:05:54.778 and then it remembers that it's here, it's stopped 124 00:05:54.778 --> 00:05:57.999 right before line five. Right after line four 125 00:05:57.999 --> 00:05:59.254 there in the middle. 126 00:05:59.254 --> 00:06:02.052 Next time you call the function, it will continue, 127 00:06:02.052 --> 00:06:04.187 so what it'll do is it will increment i by one, 128 00:06:04.187 --> 00:06:06.925 it'll rerun the loop and yield one, 129 00:06:06.925 --> 00:06:09.664 give you one and then stop. 130 00:06:09.664 --> 00:06:11.105 Next time you call it, it will continue 131 00:06:11.105 --> 00:06:12.580 increment i buy one and repeat the loop 132 00:06:12.580 --> 00:06:16.994 and then give you two, and then it will stop, and so forth. 133 00:06:16.994 --> 00:06:18.994 Eventually when you run out of numbers, 134 00:06:18.994 --> 00:06:21.077 it will yield none. 135 00:06:23.335 --> 00:06:26.418 And then you know that it's finished. 136 00:06:27.571 --> 00:06:29.582 So, how are we gonna do this? 137 00:06:29.582 --> 00:06:33.499 Let's, first of all, print out hundred numbers. 138 00:06:35.532 --> 00:06:37.700 Notice how it no longer gives us a list, 139 00:06:37.700 --> 00:06:40.020 now it gives us a generator object, 140 00:06:40.020 --> 00:06:42.040 which is called hundred numbers, 141 00:06:42.040 --> 00:06:43.451 thus the name of our function. 142 00:06:43.451 --> 00:06:47.800 It also gives us the memory address of our object. 143 00:06:47.800 --> 00:06:50.104 Notice how we defined a function, 144 00:06:50.104 --> 00:06:52.903 but the print is telling us this is an object, 145 00:06:52.903 --> 00:06:54.466 because Python in the background realises 146 00:06:54.466 --> 00:06:56.708 that you've used yield, you're making a generator, 147 00:06:56.708 --> 00:07:00.257 and it turns those into a python object. 148 00:07:00.257 --> 00:07:02.451 I'll tell you how to make these objects yourself 149 00:07:02.451 --> 00:07:05.618 without a function later on. 150 00:07:08.476 --> 00:07:13.476 So, let's store our generator in a variable. 151 00:07:14.657 --> 00:07:16.456 We're gonna call it "g", that's a fairly popular 152 00:07:16.456 --> 00:07:19.060 name for a generator in a small programme. 153 00:07:19.060 --> 00:07:20.737 If you have many generators of course you'll have to 154 00:07:20.737 --> 00:07:23.488 go for a more descriptive name. 155 00:07:23.488 --> 00:07:26.204 Then we're going to print the next value that 156 00:07:26.204 --> 00:07:28.504 the generator would give us. 157 00:07:28.504 --> 00:07:31.685 Notice that when you initialise a generator, like so, 158 00:07:31.685 --> 00:07:34.864 it doesn't run the function, it starts at the very top 159 00:07:34.864 --> 00:07:36.781 without running at all. 160 00:07:37.677 --> 00:07:40.750 When you ask it for the next value, it will 161 00:07:40.750 --> 00:07:42.490 run up until the yield and give you 162 00:07:42.490 --> 00:07:45.368 whatever value you've yielded. 163 00:07:45.368 --> 00:07:47.238 The way you ask it for the next value, is you don't 164 00:07:47.238 --> 00:07:51.661 run the function again, but you say next of "g". 165 00:07:51.661 --> 00:07:55.514 So next is a built in function that affects generators, 166 00:07:55.514 --> 00:07:58.679 and it tells it to go up to the yield. 167 00:07:58.679 --> 00:08:03.679 Okay, so if we run this, you can see now we get zero, 168 00:08:03.892 --> 00:08:07.820 because that's the very first number yielded. 169 00:08:07.820 --> 00:08:11.153 Now our generator, "g", has stopped here 170 00:08:12.992 --> 00:08:15.557 at the line 4.5 171 00:08:15.557 --> 00:08:20.557 between 4 and 5, so if we ask it for another number, 172 00:08:21.835 --> 00:08:24.255 notice that of course, at the end of a programme 173 00:08:24.255 --> 00:08:27.129 the generator is destroyed, so if we run this programme 174 00:08:27.129 --> 00:08:30.255 again this would be zero again and this would be one now. 175 00:08:30.255 --> 00:08:32.940 It doesn't remember it across programme executions, 176 00:08:32.940 --> 00:08:35.690 that's just not how Python works. 177 00:08:36.646 --> 00:08:40.603 So now we can ask it for the first number, 178 00:08:40.603 --> 00:08:42.823 and then for the next number. 179 00:08:42.823 --> 00:08:44.652 After the first number it'll be stopped 180 00:08:44.652 --> 00:08:46.084 between four and five. 181 00:08:46.084 --> 00:08:50.073 When we say next it continues up until the next yield, 182 00:08:50.073 --> 00:08:54.573 which means repeating this loop once and gives us one. 183 00:08:56.076 --> 00:08:57.243 Okay. 184 00:08:58.171 --> 00:09:03.171 Very important, the function remembers where it stopped. 185 00:09:04.709 --> 00:09:07.852 So at any point, you can stop the function from running, 186 00:09:07.852 --> 00:09:11.510 and then you can call next in order to continue it. 187 00:09:11.510 --> 00:09:15.942 So, it's not useful only for generating numbers or whatever, 188 00:09:15.942 --> 00:09:19.000 it's also useful when you want to stop a function 189 00:09:19.000 --> 00:09:21.185 from running temporarily, and then you can 190 00:09:21.185 --> 00:09:22.901 continue running it whenever you want, 191 00:09:22.901 --> 00:09:24.734 by calling next on it. 192 00:09:25.948 --> 00:09:28.217 So that's maybe not something that makes too much sense 193 00:09:28.217 --> 00:09:31.724 just now, but just remember that for when we will need it 194 00:09:31.724 --> 00:09:35.404 when we do a synchronised programming. 195 00:09:35.404 --> 00:09:37.710 So what about the range function? 196 00:09:37.710 --> 00:09:39.824 I said earlier that this was explaining essentially 197 00:09:39.824 --> 00:09:44.626 how the range function works on the inside, and that's true, 198 00:09:44.626 --> 00:09:45.953 but the range function doesn't 199 00:09:45.953 --> 00:09:48.810 give you a generator directly. 200 00:09:48.810 --> 00:09:52.029 Here, let's do something like 201 00:09:52.029 --> 00:09:55.541 "my range object", 202 00:09:55.541 --> 00:09:57.458 it's gonna be range 10. 203 00:09:58.951 --> 00:10:02.517 This behaves very much like a generator, 204 00:10:02.517 --> 00:10:05.271 but if you call next of "my range object" 205 00:10:05.271 --> 00:10:08.188 you get an error, and look at that. 206 00:10:09.355 --> 00:10:13.253 That's because the range object isn't quite a generator 207 00:10:13.253 --> 00:10:15.077 even though it behaves in this way, 208 00:10:15.077 --> 00:10:17.894 it doesn't generate a list and then allows you 209 00:10:17.894 --> 00:10:22.557 to iterate over it, it behaves in this generation-ish manner 210 00:10:22.557 --> 00:10:24.307 to be more efficient. 211 00:10:25.696 --> 00:10:27.863 "My range object", though, 212 00:10:29.150 --> 00:10:31.507 you can remember that we 213 00:10:31.507 --> 00:10:33.874 printed it out much earlier on in the course 214 00:10:33.874 --> 00:10:38.874 and we received some weird range zero to ten thing. 215 00:10:39.460 --> 00:10:43.793 That's just the "repr" function being called, 216 00:10:44.657 --> 00:10:48.381 and as you can see if we print out the "repr" outcome, 217 00:10:48.381 --> 00:10:50.958 we also see range of zero to ten. 218 00:10:50.958 --> 00:10:54.109 So "my range object" is something that behaves 219 00:10:54.109 --> 00:10:56.165 very much like this, we're gonna learn in a couple 220 00:10:56.165 --> 00:11:00.832 of videos time exactly how we can iterate over an object 221 00:11:02.133 --> 00:11:05.321 in order to make it behave like a range object. 222 00:11:05.321 --> 00:11:09.671 So, we've learned about these generators, and one more 223 00:11:09.671 --> 00:11:12.056 thing I wanted to mention actually, about the generators 224 00:11:12.056 --> 00:11:17.056 is the list function can turn a generator into a list. 225 00:11:17.475 --> 00:11:19.748 So we can print list of "g". 226 00:11:19.748 --> 00:11:22.125 Right here what we've called "next" then "next" again, 227 00:11:22.125 --> 00:11:23.458 and then "list". 228 00:11:24.423 --> 00:11:28.852 So now we have zero and one printed out for those are 229 00:11:28.852 --> 00:11:31.125 the two things in our "next" functions, 230 00:11:31.125 --> 00:11:33.668 and then we've generated a list of the generator, 231 00:11:33.668 --> 00:11:37.677 but that list of course continues from where we left off. 232 00:11:37.677 --> 00:11:40.122 The generator itself remembers what it was, 233 00:11:40.122 --> 00:11:44.238 so then the list starts from two, not from zero. 234 00:11:44.238 --> 00:11:46.893 So, it's really important to remember in generators, 235 00:11:46.893 --> 00:11:48.832 they remember where they were. 236 00:11:48.832 --> 00:11:53.832 So, when you create one, that's it from essentially 237 00:11:53.952 --> 00:11:55.645 the moment you start using it, the moment you start 238 00:11:55.645 --> 00:11:58.169 calling "next" on it, it remembers where it is, 239 00:11:58.169 --> 00:12:01.497 and you can't go backwards. Okay. 240 00:12:01.497 --> 00:12:03.225 Now, in the next few videos, we're going to learn 241 00:12:03.225 --> 00:12:04.988 more about generators. 242 00:12:04.988 --> 00:12:06.634 We're going to learn about generator classes, 243 00:12:06.634 --> 00:12:09.856 how you can create a class that behaves like a generator, 244 00:12:09.856 --> 00:12:13.238 and we're also going to learn about iterators and iterables 245 00:12:13.238 --> 00:12:15.402 in the Python, and the range objectives 246 00:12:15.402 --> 00:12:17.069 is one such example. 247 00:12:18.160 --> 00:12:19.555 Then we're going to move onto some 248 00:12:19.555 --> 00:12:21.532 functional programming examples, 249 00:12:21.532 --> 00:12:25.282 like filter, map, any and all, and enumerate. 250 00:12:26.150 --> 00:12:27.635 So, pretty excited for this section, 251 00:12:27.635 --> 00:12:30.574 I'm confident that generators are going to be something 252 00:12:30.574 --> 00:12:33.320 that you're going to need in your Python development, 253 00:12:33.320 --> 00:12:36.482 particularly as we move on further in the course. 254 00:12:36.482 --> 00:12:39.090 So with that said, I'll see you on the next video.