WEBVTT 1 00:00:00.080 --> 00:00:01.340 Hi, and welcome back! 2 00:00:01.340 --> 00:00:03.230 In this video we're going to be looking at 3 00:00:03.230 --> 00:00:04.830 our next project. 4 00:00:04.830 --> 00:00:07.800 Which is going to be a similar scraping project, 5 00:00:07.800 --> 00:00:10.360 only using a more complex site, 6 00:00:10.360 --> 00:00:12.810 and our project is going to do a few more things. 7 00:00:13.980 --> 00:00:15.830 The first thing to do is to think of 8 00:00:15.830 --> 00:00:17.400 what we want to scrape, 9 00:00:17.400 --> 00:00:19.920 what we want to get out of the site. 10 00:00:19.920 --> 00:00:20.990 And in this case, 11 00:00:20.990 --> 00:00:23.030 our project is going to do a couple of things. 12 00:00:23.030 --> 00:00:25.280 But, essentially, we're going to scrape 13 00:00:25.280 --> 00:00:27.960 this books website. 14 00:00:27.960 --> 00:00:29.580 And again, the URL is: 15 00:00:29.580 --> 00:00:32.465 books.toscrape.com 16 00:00:32.465 --> 00:00:35.400 The toscrape.com site is a collection of websites 17 00:00:35.400 --> 00:00:37.410 that we can scrape freely, 18 00:00:37.410 --> 00:00:39.430 and that have been designed for scraping. 19 00:00:39.430 --> 00:00:41.910 So they're good things to try out. 20 00:00:43.630 --> 00:00:45.220 And, what we're gonna do in our app 21 00:00:45.220 --> 00:00:47.660 is we're going to go through all the books 22 00:00:47.660 --> 00:00:48.870 they've got in their collection, 23 00:00:48.870 --> 00:00:50.900 and we're going to, sort of, 24 00:00:50.900 --> 00:00:52.940 filter through them, we're gonna see 25 00:00:52.940 --> 00:00:55.180 which ones we can buy based on price, 26 00:00:55.180 --> 00:00:56.120 and the number of views, 27 00:00:56.120 --> 00:00:57.730 and things like that. 28 00:00:57.730 --> 00:01:00.730 Essentially, making a faster 29 00:01:00.730 --> 00:01:02.410 and more efficient way 30 00:01:02.410 --> 00:01:04.020 of navigating the collection 31 00:01:04.020 --> 00:01:06.510 in order to find the books that we want to buy. 32 00:01:08.550 --> 00:01:09.480 So, as you can see, 33 00:01:09.480 --> 00:01:11.420 this is the website here on the left, 34 00:01:11.420 --> 00:01:13.680 and we've got the inspector on the right. 35 00:01:13.680 --> 00:01:15.240 If you want to open the inspector, 36 00:01:15.240 --> 00:01:18.540 you can right-click anywhere and press "Inspect." 37 00:01:18.540 --> 00:01:19.880 And, what's happening here is 38 00:01:19.880 --> 00:01:22.830 that we have 1,000 books in the catalogue 39 00:01:22.830 --> 00:01:24.450 or the collection, 40 00:01:24.450 --> 00:01:27.720 and each book is in this format, as we can see. 41 00:01:27.720 --> 00:01:28.855 We've looked at this before 42 00:01:28.855 --> 00:01:30.450 at the start of the section. 43 00:01:30.450 --> 00:01:33.700 We sort of investigated how to extract data 44 00:01:33.700 --> 00:01:36.370 if we had one of these books only. 45 00:01:36.370 --> 00:01:37.600 We're gonna take it a bit further now 46 00:01:37.600 --> 00:01:39.830 and scrape the entire page. 47 00:01:39.830 --> 00:01:41.550 So we've got all of these books 48 00:01:41.550 --> 00:01:42.780 in this format, 49 00:01:43.810 --> 00:01:46.160 and if we go to the inspector 50 00:01:46.160 --> 00:01:48.870 and we look at the HTML, we can start 51 00:01:48.870 --> 00:01:51.750 by finding out the body of the page. 52 00:01:52.950 --> 00:01:54.600 There we've got a div which contains 53 00:01:54.600 --> 00:01:56.660 everything except the header, 54 00:01:56.660 --> 00:01:58.070 and then there's a page inner 55 00:01:58.070 --> 00:02:00.220 that has essentially all the content 56 00:02:00.220 --> 00:02:01.940 and also adds sort of a bit of 57 00:02:01.940 --> 00:02:03.140 margins around the corners. 58 00:02:03.140 --> 00:02:04.280 You can see that the, 59 00:02:04.280 --> 00:02:06.000 this div here 60 00:02:06.000 --> 00:02:07.470 is slightly bigger than the other one, 61 00:02:07.470 --> 00:02:08.670 so all that's happening there 62 00:02:08.670 --> 00:02:10.490 is we're constraining the content 63 00:02:10.490 --> 00:02:12.190 so that it displays a bit nicer. 64 00:02:13.060 --> 00:02:15.070 Then, we've got the breadcrumbs, 65 00:02:15.070 --> 00:02:17.970 we've got the row here, 66 00:02:17.970 --> 00:02:20.680 and then we have the sidebar, 67 00:02:20.680 --> 00:02:22.440 which has the different topics. 68 00:02:22.440 --> 00:02:25.280 This is something we could scrape if we wanted. 69 00:02:25.280 --> 00:02:28.600 And then we're going into the book collection. 70 00:02:29.500 --> 00:02:31.850 So, in here we have a few things 71 00:02:31.850 --> 00:02:34.830 such as the different results that we've got 72 00:02:34.830 --> 00:02:36.500 and then there's a section, 73 00:02:36.500 --> 00:02:39.740 and in this section we have a div, 74 00:02:39.740 --> 00:02:42.584 and that div is the one that contains a 75 00:02:42.584 --> 00:02:44.420 "ol" class row, 76 00:02:44.420 --> 00:02:47.660 this is an unordered list of the different books 77 00:02:47.660 --> 00:02:50.060 and each book is inside 78 00:02:50.060 --> 00:02:51.830 a list item element. 79 00:02:51.830 --> 00:02:53.870 You can see the visual repeats itself 80 00:02:53.870 --> 00:02:56.460 because each book is inside a list item. 81 00:02:57.600 --> 00:02:59.330 Notice how the books are not formatted 82 00:02:59.330 --> 00:03:01.730 as list items, but nonetheless, 83 00:03:01.730 --> 00:03:04.180 with HTML you can essentially 84 00:03:04.180 --> 00:03:07.340 display anything you want in any element. 85 00:03:07.340 --> 00:03:08.970 So in this case, they're using list items, 86 00:03:08.970 --> 00:03:10.430 maybe they thought that was easier 87 00:03:10.430 --> 00:03:11.920 for development purposes. 88 00:03:11.920 --> 00:03:14.370 Normally, list items would be like bullet points. 89 00:03:15.610 --> 00:03:18.090 Now, inside the list item 90 00:03:18.090 --> 00:03:20.320 is the stuff we already know. 91 00:03:20.320 --> 00:03:22.530 An article, which is the "product_pod" 92 00:03:22.530 --> 00:03:24.010 that we've seen before, 93 00:03:24.010 --> 00:03:26.520 then we've got a div which contains the image, 94 00:03:26.520 --> 00:03:28.160 we've got the star-rating, 95 00:03:28.160 --> 00:03:29.110 the title, 96 00:03:29.110 --> 00:03:30.980 we've got the price in there, 97 00:03:30.980 --> 00:03:33.050 as well as the "add to basket" button. 98 00:03:34.240 --> 00:03:35.710 So, what we're gonna do is, 99 00:03:35.710 --> 00:03:38.570 we're going to, again, create a parser 100 00:03:38.570 --> 00:03:40.260 for the books 101 00:03:40.260 --> 00:03:42.700 that is going to be each list item 102 00:03:42.700 --> 00:03:44.010 is gonna be passed into it, 103 00:03:44.010 --> 00:03:45.510 and each parser is going to 104 00:03:45.510 --> 00:03:47.610 search for the information about that book 105 00:03:47.610 --> 00:03:48.760 within itself. 106 00:03:50.100 --> 00:03:50.933 Okay. 107 00:03:50.933 --> 00:03:51.766 So, it's gonna be essentially 108 00:03:51.766 --> 00:03:54.340 a copy of the last project 109 00:03:54.340 --> 00:03:56.190 with some added functionality, 110 00:03:56.190 --> 00:03:58.610 and of course now that we have prices 111 00:03:58.610 --> 00:04:00.670 we're gonna use the regular expressions module 112 00:04:00.670 --> 00:04:03.070 to get the price out as a number, 113 00:04:03.070 --> 00:04:05.140 and we're also going to be dealing with 114 00:04:05.140 --> 00:04:08.000 getting the star rating out of there 115 00:04:08.000 --> 00:04:11.320 and then doing some sort of sorting and filtering 116 00:04:11.320 --> 00:04:12.730 using that star rating. 117 00:04:13.800 --> 00:04:15.830 Nonetheless, my advice, as usual, 118 00:04:16.820 --> 00:04:18.890 is to, now that we have implemented 119 00:04:18.890 --> 00:04:20.920 the, quote, "scraper," 120 00:04:20.920 --> 00:04:23.400 try to implement this project on your own! 121 00:04:23.400 --> 00:04:25.280 It's not gonna be easy, 122 00:04:25.280 --> 00:04:27.250 but you already know a lot of the things 123 00:04:27.250 --> 00:04:29.230 that you need in order to tackle this, 124 00:04:29.230 --> 00:04:30.640 and if you structure your project 125 00:04:30.640 --> 00:04:33.210 using the pages and scrapers, 126 00:04:33.210 --> 00:04:35.710 essentially one page for the, 127 00:04:35.710 --> 00:04:36.820 for this entire page 128 00:04:36.820 --> 00:04:38.170 and one scraper for the, 129 00:04:38.170 --> 00:04:40.510 I'm sorry, one parser for the books, 130 00:04:40.510 --> 00:04:42.140 then you should be able to tackle this 131 00:04:42.140 --> 00:04:42.973 pretty nicely. 132 00:04:42.973 --> 00:04:43.806 And you may not be able to 133 00:04:43.806 --> 00:04:45.630 do some of the more advanced things, 134 00:04:45.630 --> 00:04:47.110 but you'll be able to do some things, 135 00:04:47.110 --> 00:04:48.460 so I definitely recommend 136 00:04:48.460 --> 00:04:49.860 you give it a break 137 00:04:49.860 --> 00:04:51.550 before moving on to the next video 138 00:04:51.550 --> 00:04:53.450 and try to implement this on your own. 139 00:04:54.610 --> 00:04:55.443 Okay? 140 00:04:55.443 --> 00:04:56.700 So once you've done that, 141 00:04:56.700 --> 00:04:58.010 let's move on to the next video, 142 00:04:58.010 --> 00:04:59.930 and we'll start implementing it there. 143 00:04:59.930 --> 00:05:00.880 I'll see you there.