WEBVTT 1 00:00:00.206 --> 00:00:01.747 Hi, and welcome back. 2 00:00:01.747 --> 00:00:03.692 In this video we are going to be looking at some code 3 00:00:03.692 --> 00:00:07.580 we wrote a few videos ago and see if we can reuse it 4 00:00:07.580 --> 00:00:08.606 for what we've got here. 5 00:00:08.606 --> 00:00:11.702 The BookParser we want to create. 6 00:00:11.702 --> 00:00:14.362 This AllBooksPage that we made in the last video 7 00:00:14.362 --> 00:00:17.361 takes in the entire HTML content 8 00:00:17.361 --> 00:00:18.921 and builds a BeautifulSoup object 9 00:00:18.921 --> 00:00:21.669 so that we can then search in it. 10 00:00:21.669 --> 00:00:25.880 Then it's got a books property that searches for our locator 11 00:00:25.880 --> 00:00:30.380 and then passes each tag it finds into our BookParser. 12 00:00:32.100 --> 00:00:35.325 We mentioned that we wrote a similar BookParser earlier on 13 00:00:35.325 --> 00:00:37.271 so let's have a look at it, I'm just going to bring it in 14 00:00:37.271 --> 00:00:40.938 from my other screen just so you can see it. 15 00:00:43.267 --> 00:00:48.267 We wrote this parsed item class earlier on in this section 16 00:00:48.808 --> 00:00:50.653 which took the entire page content 17 00:00:50.653 --> 00:00:54.124 and built a BeautifulSoup object. 18 00:00:54.124 --> 00:00:58.125 And then it had a few properties like name, link, price 19 00:00:58.125 --> 00:01:01.087 and so forth that used self.soup 20 00:01:01.087 --> 00:01:04.754 in order to find something inside this page. 21 00:01:08.052 --> 00:01:10.307 And as you can see it used sort of some locators 22 00:01:10.307 --> 00:01:12.388 and it then selected things 23 00:01:12.388 --> 00:01:15.409 and it went on to extract the title 24 00:01:15.409 --> 00:01:18.338 and the link and so forth from different things. 25 00:01:18.338 --> 00:01:19.988 What you may not have noticed is that the 26 00:01:19.988 --> 00:01:22.238 content of the HTML that we 27 00:01:24.444 --> 00:01:26.352 looked at earlier on in the section 28 00:01:26.352 --> 00:01:28.913 is actually the same content that's going to come in 29 00:01:28.913 --> 00:01:32.365 to our BookParser in our other project. 30 00:01:32.365 --> 00:01:35.138 We've got here the article with class product_pod, 31 00:01:35.138 --> 00:01:37.518 we've got the <p> star-rating 32 00:01:37.518 --> 00:01:39.489 we've got an h3 with an a tag inside 33 00:01:39.489 --> 00:01:40.741 you know, it's the same thing 34 00:01:40.741 --> 00:01:44.823 we're going to get essentially the entirety of this thing 35 00:01:44.823 --> 00:01:48.823 into our BookParser, so we can pretty much reuse 36 00:01:49.665 --> 00:01:51.966 everything that we've got here. 37 00:01:51.966 --> 00:01:56.066 Except, instead of receiving the page content, 38 00:01:56.066 --> 00:01:59.042 our BookParser is going to receive the tag, 39 00:01:59.042 --> 00:02:02.786 which is already a BeautifulSoup object. 40 00:02:02.786 --> 00:02:05.369 So, lets go ahead and copy this 41 00:02:06.605 --> 00:02:09.466 into a new class, I'm going back here to my project 42 00:02:09.466 --> 00:02:12.161 I've moved the parsed item we wrote earlier on 43 00:02:12.161 --> 00:02:14.017 in this section to another screen. 44 00:02:14.017 --> 00:02:16.973 We're going to right-click on to our parser's project 45 00:02:16.973 --> 00:02:18.538 on to our parser's package, sorry, 46 00:02:18.538 --> 00:02:20.705 and create our BookParser. 47 00:02:21.636 --> 00:02:22.957 Okay? 48 00:02:22.957 --> 00:02:25.766 Then we're going to copy the parsed item 49 00:02:25.766 --> 00:02:26.737 and I'm just going to copy it 50 00:02:26.737 --> 00:02:28.968 and then we're going to make a few changes. 51 00:02:28.968 --> 00:02:32.118 So I'm going to copy the parsed item here 52 00:02:32.118 --> 00:02:36.217 and I'm going to rename it to BookParser 53 00:02:36.217 --> 00:02:39.502 and then we are going to make a few changes. 54 00:02:39.502 --> 00:02:41.580 We're going to first of all import the locators, 55 00:02:41.580 --> 00:02:45.375 so we're going to do from locators.book_locators import 56 00:02:45.375 --> 00:02:49.224 BookLocators and then we're going to replace all instances 57 00:02:49.224 --> 00:02:52.356 of the ParsedItemLocators that we no longer need 58 00:02:52.356 --> 00:02:54.273 for these BookLocators. 59 00:02:55.123 --> 00:02:57.589 Notice how previously we already dealt with using 60 00:02:57.589 --> 00:02:59.397 regular expressions to get the price out 61 00:02:59.397 --> 00:03:02.994 and things like that, so we're going to review that quickly 62 00:03:02.994 --> 00:03:04.332 but we're not going to go into too much detail 63 00:03:04.332 --> 00:03:07.643 as to how that works, because we've already seen that. 64 00:03:07.643 --> 00:03:09.430 And instead of the pages we've mentioned earlier, 65 00:03:09.430 --> 00:03:13.566 we're now going to get a sort of parent element. 66 00:03:13.566 --> 00:03:17.064 And we no longer need to use BeautifulSoup here 67 00:03:17.064 --> 00:03:19.572 because the parent is already a BeautifulSoup object. 68 00:03:19.572 --> 00:03:22.497 Whenever you find something with BeautifulSoup 69 00:03:22.497 --> 00:03:24.447 the result of finding that is a 70 00:03:24.447 --> 00:03:27.104 BeautifulSoup object that you can search on so you can 71 00:03:27.104 --> 00:03:29.737 potentially find a specific object and then use that object 72 00:03:29.737 --> 00:03:32.396 to find other things and so forth 73 00:03:32.396 --> 00:03:34.670 so that's quite nice. 74 00:03:34.670 --> 00:03:37.581 I don't like self.soup as a name, I'm going to call it 75 00:03:37.581 --> 00:03:40.883 self.parent just so it's a bit more explicit 76 00:03:40.883 --> 00:03:43.518 where we're searching, we're not searching on the entire 77 00:03:43.518 --> 00:03:46.067 sort of soup of the page, that's what it normally means 78 00:03:46.067 --> 00:03:50.422 we're searching on a specific parent of this book. 79 00:03:50.422 --> 00:03:52.745 We're then going to follow up that page by allowing 80 00:03:52.745 --> 00:03:55.756 a couple of lines there and we're going to import re 81 00:03:55.756 --> 00:04:00.583 of course, to do our searching here for the price. 82 00:04:00.583 --> 00:04:01.416 Okay. 83 00:04:01.416 --> 00:04:06.416 So, once again, let's review exactly what's going on here. 84 00:04:06.544 --> 00:04:09.522 We have, first of all, our name property. 85 00:04:09.522 --> 00:04:12.560 That's going to use the BookLocators.NAME_LOCATOR 86 00:04:12.560 --> 00:04:15.338 to select one particular element to follow 87 00:04:15.338 --> 00:04:16.840 matching that locator and then 88 00:04:16.840 --> 00:04:20.090 extracting the title attribute of that. 89 00:04:22.137 --> 00:04:23.731 Okay. 90 00:04:23.731 --> 00:04:26.629 The link is going to be used, that locator again 91 00:04:26.629 --> 00:04:28.070 is going to find one and it's going to extract 92 00:04:28.070 --> 00:04:30.399 the href attribute of that, so notice how 93 00:04:30.399 --> 00:04:35.314 both the name and the link are finding the same tag 94 00:04:35.314 --> 00:04:37.607 and that tag happens to have two attributes. 95 00:04:37.607 --> 00:04:40.163 Title, which is the title of the book 96 00:04:40.163 --> 00:04:42.496 and href, which is the link, 97 00:04:44.520 --> 00:04:48.378 the link that takes us to the book page 98 00:04:48.378 --> 00:04:49.947 with some more details and things like that. 99 00:04:49.947 --> 00:04:52.029 Just in case you were interested. 100 00:04:52.029 --> 00:04:54.656 The price is then going to find that <p> tag 101 00:04:54.656 --> 00:04:58.131 that has the price in this format 102 00:04:58.131 --> 00:05:00.245 and it's going to extract it. 103 00:05:00.245 --> 00:05:01.396 The assumption here is that there's 104 00:05:01.396 --> 00:05:03.488 always going to be a # symbol in front 105 00:05:03.488 --> 00:05:04.508 and that there's always going to be 106 00:05:04.508 --> 00:05:07.282 some sort of period in the middle. 107 00:05:07.282 --> 00:05:09.726 That's what this regular expression syntax means, 108 00:05:09.726 --> 00:05:14.511 0-9 is a range and we're getting at least 1 of these, 109 00:05:14.511 --> 00:05:17.589 and more if there are more, the dot in the middle 110 00:05:17.589 --> 00:05:19.801 and then 0-9+. 111 00:05:19.801 --> 00:05:21.943 This is probably not going to be 112 00:05:21.943 --> 00:05:23.805 any more than two numbers at the end 113 00:05:23.805 --> 00:05:25.324 but, you know it could be more 114 00:05:25.324 --> 00:05:28.818 if they decide to do something weird. 115 00:05:28.818 --> 00:05:30.644 Then we're going to search and that's going to give us 116 00:05:30.644 --> 00:05:32.890 two groups, one group that matches 117 00:05:32.890 --> 00:05:34.135 the entirety of the string, 118 00:05:34.135 --> 00:05:37.381 another group that matches whatever is in the brackets. 119 00:05:37.381 --> 00:05:39.216 That's the group we're interested in because whatever is 120 00:05:39.216 --> 00:05:43.311 in the brackets is going to be essentially this: 121 00:05:43.311 --> 00:05:44.811 51.77 as a string. 122 00:05:46.787 --> 00:05:50.079 And then we use the float function to turn this into 123 00:05:50.079 --> 00:05:52.787 a float that will look like that. 124 00:05:52.787 --> 00:05:55.050 We can then use that to do things like sorting, 125 00:05:55.050 --> 00:05:58.580 if we want to find the cheapest books, for example. 126 00:05:58.580 --> 00:06:02.930 Finally, the rating is going to find that <p> tag 127 00:06:02.930 --> 00:06:06.106 with a rating locator and it is going to select it 128 00:06:06.106 --> 00:06:08.512 it's going to get the class attribute from it 129 00:06:08.512 --> 00:06:11.030 that contains these two attributes, 130 00:06:11.030 --> 00:06:14.028 star-rating and also a word 131 00:06:14.028 --> 00:06:16.109 representing the number of stars in the book 132 00:06:16.109 --> 00:06:20.696 so that could be four, it could be three, it could be two. 133 00:06:20.696 --> 00:06:23.829 Then we're going to get each of the classes in here 134 00:06:23.829 --> 00:06:26.662 but only if they're not equal to the star-rating class. 135 00:06:26.662 --> 00:06:28.641 So, presumably, that's going to be a list of 136 00:06:28.641 --> 00:06:31.855 a single item which is this string three, 137 00:06:31.855 --> 00:06:34.983 and we're going to return that very first item. 138 00:06:34.983 --> 00:06:35.816 Okay? 139 00:06:36.986 --> 00:06:38.720 I also mentioned earlier on in the course that you could 140 00:06:38.720 --> 00:06:41.164 use a filter function here if you prefer, 141 00:06:41.164 --> 00:06:43.709 that's something you can do as well. 142 00:06:43.709 --> 00:06:46.583 Now, something that I'm not a big fan of is returning 143 00:06:46.583 --> 00:06:49.350 this string three as a rating. 144 00:06:49.350 --> 00:06:51.408 I'd much rather return a number 145 00:06:51.408 --> 00:06:54.257 representing the number of stars it's got, 146 00:06:54.257 --> 00:06:56.501 just because when you're working with numbers 147 00:06:56.501 --> 00:06:58.694 it's much easier to do sorting, 148 00:06:58.694 --> 00:06:59.919 and if we wanted to find, for example 149 00:06:59.919 --> 00:07:02.400 all the books that are five stars 150 00:07:02.400 --> 00:07:05.324 or all the books that are over four stars, 151 00:07:05.324 --> 00:07:07.424 it's going to be easier if we've got a number 152 00:07:07.424 --> 00:07:10.750 as opposed to a string, sorting with strings is not so easy. 153 00:07:10.750 --> 00:07:13.604 So, what we're going to do is we are going to go to the top 154 00:07:13.604 --> 00:07:16.039 of the BookParser and we're going to create a static 155 00:07:16.039 --> 00:07:21.039 or class property, which is going to be called RATINGS. 156 00:07:21.064 --> 00:07:24.124 And any class property, all that means is that 157 00:07:24.124 --> 00:07:26.387 it doesn't live inside itself 158 00:07:26.387 --> 00:07:28.595 so it's not like self.ratings 159 00:07:28.595 --> 00:07:32.779 it lives inside the class, so you cannot change it 160 00:07:32.779 --> 00:07:35.743 by doing self.ratings equals something else 161 00:07:35.743 --> 00:07:40.261 and we'll access it by doing BookParser.RATINGS 162 00:07:40.261 --> 00:07:42.291 and this is going to be just what you'd expect 163 00:07:42.291 --> 00:07:44.874 one matches to one, two to two, 164 00:07:46.975 --> 00:07:49.002 three to three, 165 00:07:49.002 --> 00:07:51.502 four to four and five to five. 166 00:07:53.837 --> 00:07:57.588 Okay, now we've defined this class property 167 00:07:57.588 --> 00:08:01.452 I'm going to show you how to use it and just, just now. 168 00:08:01.452 --> 00:08:03.955 And all this does is a dictionary that 169 00:08:03.955 --> 00:08:06.916 if you look for the key one, 170 00:08:06.916 --> 00:08:09.332 it's going to give you the number one. 171 00:08:09.332 --> 00:08:11.887 And so forth for every other number. 172 00:08:11.887 --> 00:08:15.374 Now, instead of returning rating_classes[0], 173 00:08:15.374 --> 00:08:18.114 what we can return is something else. 174 00:08:18.114 --> 00:08:20.634 We can say rating is, 175 00:08:20.634 --> 00:08:22.217 or rating_number is 176 00:08:26.997 --> 00:08:30.809 BookParser.RATINGS, this is how you access 177 00:08:30.809 --> 00:08:33.817 the RATINGS thing inside BookParser. 178 00:08:33.817 --> 00:08:37.261 The same way you would access a particular function 179 00:08:37.261 --> 00:08:39.486 and BookParser.RATINGS and then 180 00:08:39.486 --> 00:08:41.463 in here, we're going to do 181 00:08:41.463 --> 00:08:43.380 .get rating_classes[0]. 182 00:08:46.608 --> 00:08:50.133 And we're going to return rating_number. 183 00:08:50.133 --> 00:08:54.575 So this is a dictionary BookParser.RATINGS, the get method 184 00:08:54.575 --> 00:08:58.553 of dictionary is going to return the value associated with 185 00:08:58.553 --> 00:09:02.317 whatever it is we pass it, so if this is the string three, 186 00:09:02.317 --> 00:09:04.163 it's going to go up to the dictionary 187 00:09:04.163 --> 00:09:06.580 and give us the number three. 188 00:09:07.426 --> 00:09:11.454 But if it does not find anything with this class in the 189 00:09:11.454 --> 00:09:14.328 dictionary, it's not going to return a key error, 190 00:09:14.328 --> 00:09:17.769 instead, it's going to return none, okay. 191 00:09:17.769 --> 00:09:20.186 So this is None if not found. 192 00:09:21.087 --> 00:09:21.920 Yep. 193 00:09:23.476 --> 00:09:26.437 If you wanted to return something else, other than none`` 194 00:09:26.437 --> 00:09:30.024 you can do that by adding another argument to the get method 195 00:09:30.024 --> 00:09:33.077 so you can, say for example, nine. 196 00:09:33.077 --> 00:09:35.395 Which it would be a pretty silly thing to do, I think 197 00:09:35.395 --> 00:09:39.543 but if the rating class that we passed in is not found 198 00:09:39.543 --> 00:09:42.868 in our BookParser.RATINGS dictionary, then it will 199 00:09:42.868 --> 00:09:46.176 return nine, or it could return like, minus one, or 200 00:09:46.176 --> 00:09:49.278 you know, it can return hello or whatever you want. 201 00:09:49.278 --> 00:09:51.849 And if it makes it easier 202 00:09:51.849 --> 00:09:54.936 you can use it as a named argument here and 203 00:09:54.936 --> 00:09:57.736 you can say default is nine 204 00:09:57.736 --> 00:09:58.990 and this is what gets returned 205 00:09:58.990 --> 00:10:01.868 if this was not found in the dictionary. 206 00:10:01.868 --> 00:10:03.937 Because I'm not really thinking that we're 207 00:10:03.937 --> 00:10:06.395 not going to be able to find anything in the dictionary 208 00:10:06.395 --> 00:10:09.333 we're going to leave it as none for the default 209 00:10:09.333 --> 00:10:11.771 but we'll see what happens as we scrape the side 210 00:10:11.771 --> 00:10:14.891 we will see if we find anything weird. 211 00:10:14.891 --> 00:10:15.724 And that's it. 212 00:10:15.724 --> 00:10:19.312 We have now been able to create our BookParser. 213 00:10:19.312 --> 00:10:20.873 We can go back to our AllBooksPage 214 00:10:20.873 --> 00:10:22.864 and make sure to import it 215 00:10:22.864 --> 00:10:26.614 so from parsers.book_parser import BookParser 216 00:10:28.009 --> 00:10:31.912 and as soon as we pass in the page content to this page 217 00:10:31.912 --> 00:10:33.942 we'll be able to access the books property 218 00:10:33.942 --> 00:10:37.109 which is going to find all the li tags 219 00:10:37.994 --> 00:10:40.726 and pass them into the BookParser. 220 00:10:40.726 --> 00:10:43.800 Then we'll have a list of BookParser objects and 221 00:10:43.800 --> 00:10:46.025 for each object we'll be able to do things like 222 00:10:46.025 --> 00:10:50.589 access the name property or the link and so forth. 223 00:10:50.589 --> 00:10:52.380 Hopefully this makes sense, let's go over to 224 00:10:52.380 --> 00:10:53.972 the next video and we're going to be writing 225 00:10:53.972 --> 00:10:56.012 a little bit of code in our app.py 226 00:10:56.012 --> 00:11:00.082 just to use this page and extract some information. 227 00:11:00.082 --> 00:11:01.556 I'll see you in the next video.