WEBVTT 1 00:00:00.090 --> 00:00:01.260 Hi and welcome back. 2 00:00:01.260 --> 00:00:05.050 Let's make our first request with aiohttp. 3 00:00:05.960 --> 00:00:09.400 Here in this project I'm just going to create a new file 4 00:00:09.400 --> 00:00:13.050 just for testing aiohttp and it's gonna be called 5 00:00:13.050 --> 00:00:15.290 asynch_request.py. 6 00:00:15.290 --> 00:00:18.240 By the way the code for this is linked in lecture 7 00:00:18.240 --> 00:00:21.553 and is available, maybe in a slightly different setup, 8 00:00:21.553 --> 00:00:24.350 but nonetheless there's gonna be samples folder 9 00:00:24.350 --> 00:00:26.340 for you to look at these codes that we're gonna 10 00:00:26.340 --> 00:00:28.300 write over these lectures and then of course 11 00:00:28.300 --> 00:00:30.260 at the end as well when we turn the full thing 12 00:00:30.260 --> 00:00:31.660 into an asynchronous scrapper, 13 00:00:31.660 --> 00:00:33.410 the full code is available as well. 14 00:00:34.880 --> 00:00:37.795 So in order to make our first web request, 15 00:00:37.795 --> 00:00:40.300 the first thing we need to do is of course, 16 00:00:40.300 --> 00:00:43.560 is to instal aiohttp. 17 00:00:43.560 --> 00:00:45.860 Now soon we're going to look at 18 00:00:46.900 --> 00:00:49.960 how to handle dependencies in Python projects a bit better, 19 00:00:49.960 --> 00:00:52.950 but for now we're going to stick with installing things 20 00:00:52.950 --> 00:00:56.680 directly in PyCharm by doing, you know, 21 00:00:56.680 --> 00:01:00.470 going over to our project, Project Interpreter, 22 00:01:00.470 --> 00:01:03.930 and there with plus installing aiohttp. 23 00:01:06.000 --> 00:01:08.250 The current version that I am using is 3.0.5. 24 00:01:09.260 --> 00:01:11.180 this maybe a different when you're using it, 25 00:01:11.180 --> 00:01:14.311 but don't worry, if there are any breaking changes 26 00:01:14.311 --> 00:01:17.294 we will re-record with a new version and soon. 27 00:01:17.294 --> 00:01:20.940 So if you instal this and you notice that 28 00:01:20.940 --> 00:01:22.440 things don't work quite the way we expect 29 00:01:22.440 --> 00:01:24.230 just say in the course Q and A 30 00:01:24.230 --> 00:01:26.230 and we will re-record as soon as possible. 31 00:01:26.230 --> 00:01:28.250 Okay, just instal this package 32 00:01:28.250 --> 00:01:29.957 and this is all we need to instal. 33 00:01:29.957 --> 00:01:33.090 And so when that's installed you can close this window 34 00:01:35.200 --> 00:01:36.980 and we can begin using it. 35 00:01:38.360 --> 00:01:39.960 So that's it, done. 36 00:01:39.960 --> 00:01:44.860 We can press OK and now we can import aiohttp. 37 00:01:46.230 --> 00:01:50.760 So let's first start with fetching a single page. 38 00:01:50.760 --> 00:01:53.250 The first thing we need to do is create the core routine 39 00:01:53.250 --> 00:01:54.890 and this is like a generator 40 00:01:54.890 --> 00:01:58.030 that can suspend and resume their execution at any time 41 00:01:58.030 --> 00:01:59.880 by using a wait. 42 00:01:59.880 --> 00:02:02.880 So we're gonna do async def fetch_page. 43 00:02:04.179 --> 00:02:05.190 And what we're gonna pass through this function 44 00:02:05.190 --> 00:02:06.990 is the URL that we want to fetch. 45 00:02:08.270 --> 00:02:10.190 Now the next thing we do, 46 00:02:10.190 --> 00:02:15.000 is we must tell aiohttp to create a client session. 47 00:02:15.000 --> 00:02:17.670 A client session just essentially creates 48 00:02:17.670 --> 00:02:20.560 a bunch of connections and puts them in a pool, 49 00:02:20.560 --> 00:02:22.270 pretty much like a thread pool, 50 00:02:22.270 --> 00:02:25.040 except it's a connection pool, so that we can reuse them 51 00:02:25.040 --> 00:02:27.710 without having to create new connections each time. 52 00:02:27.710 --> 00:02:30.290 So what we're gonna do is a bit of a new bit of syntax. 53 00:02:30.290 --> 00:02:34.940 Async_with aiohttp.ClientSession as session. 54 00:02:38.796 --> 00:02:41.050 All that's happening here is 55 00:02:41.050 --> 00:02:43.565 we are doing a normal context manager, 56 00:02:43.565 --> 00:02:45.800 we're creating this new ClientSession 57 00:02:45.800 --> 00:02:49.320 which is a connection pool and also a cookie storage. 58 00:02:49.320 --> 00:02:51.880 We're gonna talk more about that soon. 59 00:02:52.870 --> 00:02:57.325 We're calling it session, but because it's async, 60 00:02:57.325 --> 00:03:00.332 what happens in an async context manager 61 00:03:00.332 --> 00:03:04.380 is that it can have yield in there enter 62 00:03:04.380 --> 00:03:06.530 or in the exit methods or in both. 63 00:03:06.530 --> 00:03:08.830 So essentially it's just a way of potentially 64 00:03:08.830 --> 00:03:12.220 suspending the execution of the context manager 65 00:03:12.220 --> 00:03:14.590 when it starts or suspending it when it ends 66 00:03:14.590 --> 00:03:15.510 and resuming it. 67 00:03:15.510 --> 00:03:20.510 Just adds like a new point where it can suspend and resume 68 00:03:21.240 --> 00:03:22.530 just in case you want to do something 69 00:03:22.530 --> 00:03:25.420 that was heavy duty inside the context manager, 70 00:03:25.420 --> 00:03:27.200 allows you to suspend and resume 71 00:03:27.200 --> 00:03:28.980 at the start or at the end of the context manager. 72 00:03:28.980 --> 00:03:32.210 That's the only difference with the normal context manager 73 00:03:32.210 --> 00:03:33.710 and the async context manager. 74 00:03:35.270 --> 00:03:39.740 For more information about this, instead of enter and exit, 75 00:03:43.220 --> 00:03:47.300 the new methods are aenter and aexit. 76 00:03:49.480 --> 00:03:51.530 Okay, we're not gonna get into too much depth 77 00:03:51.530 --> 00:03:53.680 about creating your own asynchronous context managers, 78 00:03:53.680 --> 00:03:56.370 just now that if you wanna create an asynchronous manager 79 00:03:56.370 --> 00:04:01.095 well you can do a wait or yield inside it, 80 00:04:01.095 --> 00:04:03.300 and you have to use aenter and aexit 81 00:04:03.300 --> 00:04:04.600 instead of enter and exit. 82 00:04:05.570 --> 00:04:07.730 Now that just a bit of an aside there. 83 00:04:07.730 --> 00:04:11.100 This we're going to do async with session.get 84 00:04:11.100 --> 00:04:14.040 and we're gonna get the URL as response. 85 00:04:14.040 --> 00:04:16.900 And then we're gonna return the response.status code. 86 00:04:18.450 --> 00:04:20.920 This can be our first task. 87 00:04:20.920 --> 00:04:24.876 This is a core routine, that first of all creates a session, 88 00:04:24.876 --> 00:04:28.790 then it sort of gets the URL we passed in. 89 00:04:28.790 --> 00:04:31.377 This asks the server for the contents, 90 00:04:31.377 --> 00:04:35.540 calls a response, and then returns the response.status. 91 00:04:35.540 --> 00:04:37.450 But of course before responding, 92 00:04:37.450 --> 00:04:38.990 before returning the response.status, 93 00:04:38.990 --> 00:04:43.990 it can potentially be suspended here or here, 94 00:04:44.390 --> 00:04:47.350 when it starts or when it ends. 95 00:04:47.350 --> 00:04:49.598 So when we can, for example, 96 00:04:49.598 --> 00:04:53.620 get the URL or send the request to the server 97 00:04:53.620 --> 00:04:56.950 and then suspend while we wait for data to come back. 98 00:04:56.950 --> 00:05:00.010 When the data comes back, we'll be at the exit, 99 00:05:00.010 --> 00:05:03.189 sorry, at the end of the enter section of the generator 100 00:05:03.189 --> 00:05:05.130 and we can continue execution 101 00:05:05.130 --> 00:05:06.590 and then we can get this status. 102 00:05:06.590 --> 00:05:10.340 And this is not asynchronous, this is just a normal return. 103 00:05:10.340 --> 00:05:12.350 So then the function would end there. 104 00:05:13.743 --> 00:05:15.850 Now the only purpose of asyncio 105 00:05:15.850 --> 00:05:19.410 and these asynchronous functions is to potentially suspend 106 00:05:19.410 --> 00:05:22.800 execution at any point and resume it after. 107 00:05:22.800 --> 00:05:24.680 Placing these things in the correct places 108 00:05:24.680 --> 00:05:27.620 means that we can suspend execution when we're waiting 109 00:05:27.620 --> 00:05:31.010 and we can resume execution once we want to stop waiting. 110 00:05:34.230 --> 00:05:36.679 Now we must have a task scheduler 111 00:05:36.679 --> 00:05:41.679 that will do the .send or the next. 112 00:05:41.740 --> 00:05:45.850 Remember this just yield from, yield from 113 00:05:45.850 --> 00:05:48.830 inside of here that are waits which are yield froms, 114 00:05:48.830 --> 00:05:52.090 so we need something that is gonna do this send 115 00:05:52.090 --> 00:05:53.950 or this next. 116 00:05:53.950 --> 00:05:57.850 And now, as of Python 3.4, I believe, we have that, 117 00:05:57.850 --> 00:05:59.780 which is the asyncio library. 118 00:06:01.020 --> 00:06:02.630 If you've watched that talk, which I hope you did, 119 00:06:02.630 --> 00:06:05.560 you will have seen asyncio mentioned a couple of times. 120 00:06:05.560 --> 00:06:07.990 The asyncio library comes built in Python 121 00:06:07.990 --> 00:06:11.320 and is very small, well it's not that small, 122 00:06:11.320 --> 00:06:13.910 but it's fairly small the things we have to know 123 00:06:13.910 --> 00:06:16.520 and it allows us to schedule 124 00:06:16.520 --> 00:06:20.570 and manage these co-routines, these tasks. 125 00:06:20.570 --> 00:06:22.560 It's really straighforward to use, 126 00:06:22.560 --> 00:06:25.080 at least the way we're gonna do it right now. 127 00:06:25.080 --> 00:06:27.798 There's a bit more to use than what we cover, 128 00:06:27.798 --> 00:06:31.107 but in order to use it, 129 00:06:31.107 --> 00:06:35.150 in some ways it's fairly straighforward. 130 00:06:35.150 --> 00:06:37.675 All we do is we get an event_loop 131 00:06:37.675 --> 00:06:41.060 and then we do loop.run_until_complete(fetch_page) 132 00:06:42.830 --> 00:06:45.730 for example, google.com. 133 00:06:48.750 --> 00:06:53.730 When you call fetch_page you don't get the response.status. 134 00:06:53.730 --> 00:06:55.870 We know this already from what we've looked at earlier. 135 00:06:55.870 --> 00:06:57.690 When you call fetch_page what you get 136 00:06:57.690 --> 00:06:59.140 is a co-routine object. 137 00:07:00.086 --> 00:07:03.240 So this isn't actually giving the response, 138 00:07:03.240 --> 00:07:04.770 even though we're calling function. 139 00:07:04.770 --> 00:07:06.680 All this is doing is creating the co-routine 140 00:07:06.680 --> 00:07:11.680 in the background that you can then send or next to. 141 00:07:11.900 --> 00:07:14.210 When you do that, it sort of goes through 142 00:07:14.210 --> 00:07:16.420 each resume and response. 143 00:07:16.420 --> 00:07:18.460 So what this run_until_complete is doing 144 00:07:18.460 --> 00:07:20.610 is it's using this event_loop 145 00:07:20.610 --> 00:07:23.700 in order to continually resume this function 146 00:07:23.700 --> 00:07:24.700 until it's complete. 147 00:07:27.907 --> 00:07:29.670 So we can run this, we can right click 148 00:07:29.670 --> 00:07:31.920 the async request and run it. 149 00:07:31.920 --> 00:07:34.340 And of course we didn't really print anything, 150 00:07:34.340 --> 00:07:39.340 but we can print response.status here 151 00:07:40.070 --> 00:07:42.320 and you'll see that something does come back. 152 00:07:43.490 --> 00:07:45.560 Yep, 200, which means it's all okay. 153 00:07:45.560 --> 00:07:47.937 As you can see, this did run. 154 00:07:47.937 --> 00:07:51.940 So this is the simplest way that we can get an event_loop 155 00:07:51.940 --> 00:07:56.300 and run a task and a task is just an async function 156 00:07:56.300 --> 00:07:57.810 that has been defined. 157 00:07:57.810 --> 00:07:59.620 Okay, all we need is to get the event_loop 158 00:07:59.620 --> 00:08:01.180 and run it until it's complete. 159 00:08:03.670 --> 00:08:06.080 If you wanted to make more asynchronous requests, 160 00:08:06.080 --> 00:08:07.780 we're gonna do that just now as well. 161 00:08:07.780 --> 00:08:10.620 What we're gonna do first of all is import time 162 00:08:10.620 --> 00:08:13.530 so we can stop measuring how long this takes. 163 00:08:13.530 --> 00:08:16.770 Then before we do any of the session or anything like that, 164 00:08:16.770 --> 00:08:19.090 we're gonna do page_start is time.time. 165 00:08:20.370 --> 00:08:25.370 And right before we return, we're gonna print that 166 00:08:27.870 --> 00:08:32.170 the page took time.time minus page_start. 167 00:08:33.800 --> 00:08:38.020 I call it state, that should be start there. 168 00:08:39.140 --> 00:08:43.220 Okay if we run this again, you see that it says 169 00:08:43.220 --> 00:08:45.420 page took 0.17 seconds. 170 00:08:45.420 --> 00:08:46.840 So it was pretty quick. 171 00:08:46.840 --> 00:08:48.920 The programme as a whole took a bit longer, 172 00:08:48.920 --> 00:08:51.653 but that's because it involves creating the event_loop 173 00:08:51.653 --> 00:08:55.640 and starting new tasks, scheduling them and so forth. 174 00:08:55.640 --> 00:08:59.604 But the page request itself only took 0.17 seconds. 175 00:08:59.604 --> 00:09:03.560 If you round this using the requests library 176 00:09:03.560 --> 00:09:06.460 like we did earlier on, it would take more or less 177 00:09:06.460 --> 00:09:07.960 the same amount of time. 178 00:09:07.960 --> 00:09:10.080 Okay, there would be no difference, 179 00:09:10.080 --> 00:09:12.138 just because all we're doing here 180 00:09:12.138 --> 00:09:14.816 is running a single request. 181 00:09:14.816 --> 00:09:18.800 If you want to run multiple requests though, 182 00:09:18.800 --> 00:09:20.830 we can do that very easily. 183 00:09:20.830 --> 00:09:22.388 We'll say something like tasks is, 184 00:09:22.388 --> 00:09:24.650 and now we're gonna do a list comprehension 185 00:09:24.650 --> 00:09:28.340 where we're gonna fetch_page off google.com 186 00:09:28.340 --> 00:09:31.250 for i in range 50, let's say. 187 00:09:32.720 --> 00:09:35.826 Okay so now we've got 50 co-routine objects 188 00:09:35.826 --> 00:09:37.680 that need to be ran. 189 00:09:38.820 --> 00:09:40.410 So how we're gonna do that? 190 00:09:40.410 --> 00:09:44.170 Well we're going to say loop.run all these tasks 191 00:09:44.170 --> 00:09:46.151 until they have been completed. 192 00:09:46.151 --> 00:09:49.470 But we can only pass single task 193 00:09:49.470 --> 00:09:51.816 to the run_until_complete function. 194 00:09:51.816 --> 00:09:56.610 So asyncio comes with a pretty handy function 195 00:09:56.610 --> 00:10:00.220 that allows us to collect a bunch of tasks together 196 00:10:00.220 --> 00:10:02.440 and run them as a single task, 197 00:10:02.440 --> 00:10:04.570 so that it's a bit easier to do this 198 00:10:04.570 --> 00:10:07.650 and all that is is asyncio.gather. 199 00:10:07.650 --> 00:10:11.680 And we're gonna pass it all the tasks as arguments. 200 00:10:11.680 --> 00:10:16.680 So it will be tasks(0), tasks(1), tasks(2), tasks(3) 201 00:10:17.520 --> 00:10:19.590 and so on until 50. 202 00:10:19.590 --> 00:10:22.661 Or as you already know, we can use argument unpacking 203 00:10:22.661 --> 00:10:26.180 to do star tasks. 204 00:10:27.290 --> 00:10:31.240 Okay, now as part of this I'm also going to time this. 205 00:10:31.240 --> 00:10:34.940 So I'm gonna say start is time.time 206 00:10:34.940 --> 00:10:39.850 and end is gonna be all took time.time minus start. 207 00:10:41.050 --> 00:10:46.000 Notice how I'm only measuring the loop_run_until_complete. 208 00:10:46.000 --> 00:10:48.280 I'm not measuring the creation the tasks, 209 00:10:48.280 --> 00:10:50.108 because this is pretty much instant. 210 00:10:50.108 --> 00:10:53.889 No requests are actually happening in this line. 211 00:10:53.889 --> 00:10:56.560 Only the core routines have been created. 212 00:10:56.560 --> 00:10:58.410 Then when we stop running them, 213 00:10:58.410 --> 00:11:01.310 this does the .send to the first task 214 00:11:01.310 --> 00:11:03.530 and runs it up until the asyncio. 215 00:11:03.530 --> 00:11:06.254 Then that suspends and this moves onto the second one 216 00:11:06.254 --> 00:11:08.500 and moves it up 'til there, then this suspends 217 00:11:08.500 --> 00:11:09.900 and moves onto the third one. 218 00:11:09.900 --> 00:11:13.536 So first of all, all 50 tasks will end up here. 219 00:11:13.536 --> 00:11:15.997 Then one by one they will be moved here. 220 00:11:15.997 --> 00:11:19.370 And then one by one they will return. 221 00:11:19.370 --> 00:11:22.940 But of course, they may take different amounts of time. 222 00:11:22.940 --> 00:11:27.500 So one request to google.com may take 0.5 seconds 223 00:11:27.500 --> 00:11:28.750 if it's a very slow one. 224 00:11:28.750 --> 00:11:30.712 Another one may take 0.1 seconds. 225 00:11:30.712 --> 00:11:32.550 So as soon as you start doing this 226 00:11:32.550 --> 00:11:37.550 and you only resume execution when you are ready to do so 227 00:11:37.590 --> 00:11:40.040 you'll see that these tasks are not nescessarily 228 00:11:40.040 --> 00:11:41.970 all gonna finish in the same order. 229 00:11:41.970 --> 00:11:44.150 And okay, so let's press play. 230 00:11:45.670 --> 00:11:47.943 And as you can see, that was extremely quickly. 231 00:11:47.943 --> 00:11:52.943 We've got some tasks that took 0.32 seconds, 232 00:11:53.250 --> 00:11:56.291 and as they finish more or less in order 233 00:11:56.291 --> 00:12:00.168 we can see that at the bottom some of that took 0.46 seconds 234 00:12:00.168 --> 00:12:04.473 and in total it took 0.48 seconds. 235 00:12:04.473 --> 00:12:07.150 Not the addition of all of these things, 236 00:12:07.150 --> 00:12:10.580 but rather the longest one plus a couple of milliseconds 237 00:12:10.580 --> 00:12:12.750 just to set up the event loop 238 00:12:12.750 --> 00:12:15.923 and do the scheduling of the tasks and so forth. 239 00:12:15.923 --> 00:12:18.920 So this is amazing, 240 00:12:18.920 --> 00:12:21.810 because now instead of having to wait for each page 241 00:12:21.810 --> 00:12:24.010 to finish before we get the next one, 242 00:12:24.010 --> 00:12:26.670 we're gonna essentially just send the 50 requests out 243 00:12:26.670 --> 00:12:30.050 to Google and just wait until they come back. 244 00:12:31.870 --> 00:12:33.570 And of course that's gonna depend on 245 00:12:33.570 --> 00:12:38.550 whether Google can deal with 50 requests at the same time. 246 00:12:38.550 --> 00:12:42.800 Google is a massive company, it has a lot of servers, 247 00:12:42.800 --> 00:12:45.910 and it can handle 50 requests at the same time. 248 00:12:45.910 --> 00:12:48.430 Other websites won't be able to handle 50 requests 249 00:12:48.430 --> 00:12:49.550 at the same time and if you do that, 250 00:12:49.550 --> 00:12:51.160 you may actually break them. 251 00:12:51.160 --> 00:12:52.500 You may actually break the site 252 00:12:52.500 --> 00:12:57.500 or they may say, no you cannot do 50 requests at a time. 253 00:12:57.670 --> 00:13:00.530 I'm going to be very slow in responding to you 254 00:13:00.530 --> 00:13:04.780 in order to prevent you from breaking their server. 255 00:13:04.780 --> 00:13:07.150 So that's a couple of things you need to be careful about. 256 00:13:07.150 --> 00:13:10.003 But as you can see with Google, this is all you need 257 00:13:10.003 --> 00:13:13.961 in order to get 50 pages asynchronously. 258 00:13:13.961 --> 00:13:16.202 So again what we've done is created our task 259 00:13:16.202 --> 00:13:21.202 and using aiohttp which supports this async_with statements 260 00:13:22.156 --> 00:13:26.110 we have created our session and then used that 261 00:13:26.110 --> 00:13:29.900 which gives us a connection pool to get each URL 262 00:13:29.900 --> 00:13:31.690 that we passed in. 263 00:13:31.690 --> 00:13:34.610 And at the end we've returned a response.status code, 264 00:13:34.610 --> 00:13:37.518 that's just to get something back from there. 265 00:13:37.518 --> 00:13:39.470 And then in order to run them, 266 00:13:39.470 --> 00:13:42.997 we've got an event_loop from the asyncio framework. 267 00:13:42.997 --> 00:13:47.584 We've created all our tasks, which are just these functions 268 00:13:47.584 --> 00:13:50.460 and then we've run them all until complete 269 00:13:50.460 --> 00:13:52.740 by using asyncio.gather. 270 00:13:52.740 --> 00:13:54.910 And gather collects all our tasks 271 00:13:54.910 --> 00:13:58.900 and allows them to run as one in this run_until_complete. 272 00:13:58.900 --> 00:14:01.440 Otherwise you would have to do run_until_complete 273 00:14:01.440 --> 00:14:04.520 once for each task, which would be a bit more tedious. 274 00:14:04.520 --> 00:14:06.882 So asyncio.gather, pretty good for that. 275 00:14:06.882 --> 00:14:10.050 Now that's it for this video. 276 00:14:10.050 --> 00:14:13.090 I just wanted to show you how to go about 277 00:14:13.090 --> 00:14:14.976 making these asynchronous requests. 278 00:14:14.976 --> 00:14:18.870 Feel free to compare this with your synchronous code. 279 00:14:18.870 --> 00:14:20.820 If you open up .py by the way, 280 00:14:20.820 --> 00:14:22.660 you can see this is a synchronous code. 281 00:14:22.660 --> 00:14:25.100 So if you replace this for a google.com 282 00:14:25.100 --> 00:14:27.480 and put a couple of timing functions in here, 283 00:14:27.480 --> 00:14:28.900 you'll see how long that takes. 284 00:14:28.900 --> 00:14:33.329 It's about 10 seconds, whereas this was about 0.5 seconds. 285 00:14:33.329 --> 00:14:35.530 So now what we're gonna do is we're gonna 286 00:14:35.530 --> 00:14:36.850 just make this a bit more efficient. 287 00:14:36.850 --> 00:14:39.650 Get a couple of functions in here to deal with 288 00:14:39.650 --> 00:14:42.340 getting multiple pages more efficiently 289 00:14:42.340 --> 00:14:44.980 and then we'll move onto turning our scraper 290 00:14:44.980 --> 00:14:46.850 into an asynchronous scrape. 291 00:14:46.850 --> 00:14:48.430 I'll see you on the next video.