WEBVTT 1 00:00:00.180 --> 00:00:01.290 Hi, welcome back. 2 00:00:01.290 --> 00:00:02.730 In this video, we're looking at something 3 00:00:02.730 --> 00:00:04.030 a bit more complicated, 4 00:00:04.030 --> 00:00:07.490 understanding some slightly more complex HTML. 5 00:00:07.490 --> 00:00:09.653 Let's create a new file in our project, 6 00:00:10.550 --> 00:00:13.363 which I'm gonna call middle_html. 7 00:00:14.351 --> 00:00:16.560 By the way, the files in the project, 8 00:00:16.560 --> 00:00:18.550 which are available at the start of this section, 9 00:00:18.550 --> 00:00:19.840 also contain this exact code, 10 00:00:19.840 --> 00:00:21.120 and you can have a look at them. 11 00:00:21.120 --> 00:00:23.240 The files may be named slightly differently, 12 00:00:23.240 --> 00:00:24.840 and that should be fine, I hope. 13 00:00:25.770 --> 00:00:27.540 Once again, we're gonna do the same 14 00:00:27.540 --> 00:00:30.380 from bs4 import BeautifulSoup, 15 00:00:30.380 --> 00:00:33.210 and then, I'm going to paste in some HTML. 16 00:00:33.210 --> 00:00:35.880 In this case, our HTML is going to represent 17 00:00:35.880 --> 00:00:40.220 an item that is being on sale in an online website. 18 00:00:40.220 --> 00:00:42.300 So let's say we've managed to extract 19 00:00:42.300 --> 00:00:43.900 the content from the website. 20 00:00:43.900 --> 00:00:45.580 We're gonna look at how to do that, 21 00:00:45.580 --> 00:00:47.623 and we've ended up with this HTML here, 22 00:00:48.760 --> 00:00:50.870 and with this HTML, 23 00:00:50.870 --> 00:00:53.990 we can find out the price of the item, 24 00:00:53.990 --> 00:00:56.483 the star rating of the item, and so forth. 25 00:00:57.460 --> 00:01:00.470 Let me show you exactly what this looks like on the website. 26 00:01:00.470 --> 00:01:03.040 Then it'll make a bit more sense. 27 00:01:03.040 --> 00:01:06.010 What we've got here is a demo website 28 00:01:06.010 --> 00:01:07.950 for web scraping purposes, 29 00:01:07.950 --> 00:01:10.820 and let's say we've managed to extract 30 00:01:10.820 --> 00:01:13.500 some HTML content from it. 31 00:01:13.500 --> 00:01:16.250 The content that I've extracted and put in the file 32 00:01:16.250 --> 00:01:19.420 is the content that represents this object exactly, 33 00:01:19.420 --> 00:01:21.640 this bit of HTML here. 34 00:01:21.640 --> 00:01:24.720 It contains an image, a rating, a title, 35 00:01:24.720 --> 00:01:27.410 which seems to be partly missing, 36 00:01:27.410 --> 00:01:30.320 and the price, and then, whether it's in stock or not, 37 00:01:30.320 --> 00:01:33.253 and finally, a button to add it to the basket. 38 00:01:34.150 --> 00:01:35.610 So this HTML here 39 00:01:36.790 --> 00:01:39.840 is what we've copied into our file, 40 00:01:39.840 --> 00:01:41.980 and again, we're going to learn how to get the HTML 41 00:01:41.980 --> 00:01:44.230 from the site very soon. 42 00:01:44.230 --> 00:01:47.763 So let's go back into PyCharm and look at the HTML. 43 00:01:49.130 --> 00:01:50.600 This is what we've got. 44 00:01:50.600 --> 00:01:54.830 We've got a list item here. 45 00:01:54.830 --> 00:01:59.070 That's because each element is actually in a list 46 00:01:59.070 --> 00:02:00.440 in the website, 47 00:02:00.440 --> 00:02:03.040 so an element is a list item of a big list 48 00:02:03.040 --> 00:02:04.570 that is formatted and structured 49 00:02:04.570 --> 00:02:07.960 in a bit of a different way than you'd normally see lists. 50 00:02:07.960 --> 00:02:10.023 Then, it's got an article inside it. 51 00:02:11.020 --> 00:02:12.660 You can see that the article 52 00:02:12.660 --> 00:02:14.980 spans essentially the entire document, 53 00:02:14.980 --> 00:02:18.150 and that is everything that we're interested in. 54 00:02:18.150 --> 00:02:22.490 We've got a div who's has a class image_container, 55 00:02:22.490 --> 00:02:25.490 and a div in HTML is normally just 56 00:02:25.490 --> 00:02:28.320 something used to structure your content. 57 00:02:28.320 --> 00:02:30.080 So it's a, think of it as a rectangle 58 00:02:30.080 --> 00:02:31.720 and you can put things inside it, 59 00:02:31.720 --> 00:02:34.100 and in here, what they've put is a link, 60 00:02:34.100 --> 00:02:38.050 and that's the a, which is used for linking. 61 00:02:38.050 --> 00:02:42.520 href is the URL or the address of the page 62 00:02:42.520 --> 00:02:45.030 that the link goes to, 63 00:02:45.030 --> 00:02:48.590 and inside the link, notice how the link starts here 64 00:02:48.590 --> 00:02:51.400 and ends here in this closing tag, 65 00:02:51.400 --> 00:02:53.030 you've got a image. 66 00:02:53.030 --> 00:02:54.740 This is an image tag, 67 00:02:54.740 --> 00:02:56.810 and this is the source of the image. 68 00:02:56.810 --> 00:03:00.160 That is some JPG file. 69 00:03:00.160 --> 00:03:02.540 It's got some alt property, 70 00:03:02.540 --> 00:03:04.363 and it's got a class property. 71 00:03:05.320 --> 00:03:08.400 The alt property is the alternate text 72 00:03:08.400 --> 00:03:12.180 to show if the image cannot load for some reason. 73 00:03:12.180 --> 00:03:14.637 So that is the title of the book, actually, 74 00:03:14.637 --> 00:03:16.137 "A Light in the Attic." 75 00:03:18.370 --> 00:03:22.390 Outside the div, we have a p tag, a paragraph tag, 76 00:03:22.390 --> 00:03:23.940 which is used for the star rating. 77 00:03:23.940 --> 00:03:26.530 This is a bit of a bizarre of doing a star rating, 78 00:03:26.530 --> 00:03:30.400 but you can see that there is a class star-rating 79 00:03:30.400 --> 00:03:33.330 and another class, which is separated by a space in HTML, 80 00:03:33.330 --> 00:03:35.453 called Three with a capital T. 81 00:03:36.490 --> 00:03:38.920 So this book has three stars, 82 00:03:38.920 --> 00:03:42.210 but you can see inside it, there are five i tags, 83 00:03:42.210 --> 00:03:44.230 which are used for icons. 84 00:03:44.230 --> 00:03:48.200 So you can see that we must look at Three, this class here, 85 00:03:48.200 --> 00:03:50.700 and not at the number of i tags 86 00:03:50.700 --> 00:03:53.030 because the i tags are always five. 87 00:03:53.030 --> 00:03:54.380 So we don't wanna look at that. 88 00:03:54.380 --> 00:03:55.780 We wanna look at this Three. 89 00:03:57.300 --> 00:03:59.050 Then outside the p, you have an h3, 90 00:03:59.990 --> 00:04:01.510 which is the title of the book, 91 00:04:01.510 --> 00:04:03.820 and you can see that the contents of the h3 92 00:04:03.820 --> 00:04:07.060 are A Light in the dot dot dot. 93 00:04:07.060 --> 00:04:08.860 Presumably, the book name is too long, 94 00:04:08.860 --> 00:04:11.100 and it couldn't be displayed fully on the website. 95 00:04:11.100 --> 00:04:15.140 However, the h3 contains inside it a link 96 00:04:15.140 --> 00:04:17.660 to the book page presumably, 97 00:04:17.660 --> 00:04:21.370 and also has a title property in it, a title attribute, 98 00:04:21.370 --> 00:04:23.120 that is also the title of the book. 99 00:04:24.390 --> 00:04:26.690 Then you've got a div which is the product price, 100 00:04:26.690 --> 00:04:28.630 and inside it, you've got a p tag 101 00:04:28.630 --> 00:04:31.220 which contains the class price_color, 102 00:04:31.220 --> 00:04:33.130 and that contains the price of the book 103 00:04:33.130 --> 00:04:34.880 with a pound symbol in front of it. 104 00:04:36.040 --> 00:04:39.190 Also inside the div, you have this other p tag called, 105 00:04:39.190 --> 00:04:42.983 with class instock and another class called availability. 106 00:04:43.850 --> 00:04:45.780 Presumably, this class instock 107 00:04:45.780 --> 00:04:48.370 will only be present in the document 108 00:04:48.370 --> 00:04:49.650 when the thing is in stock, 109 00:04:49.650 --> 00:04:52.290 and if it's out of stock, it'll say something else possibly. 110 00:04:52.290 --> 00:04:53.240 We don't know that. 111 00:04:54.290 --> 00:04:56.510 Inside that p tag, you've got an i class, 112 00:04:56.510 --> 00:04:57.910 which is for an icon. 113 00:04:57.910 --> 00:04:59.780 An i tag, sorry, that's for an icon, 114 00:04:59.780 --> 00:05:02.313 and that has a class, icon-ok. 115 00:05:03.630 --> 00:05:04.920 Then you've got a text, In stock. 116 00:05:04.920 --> 00:05:07.600 Notice how this is weirdly formatted, 117 00:05:07.600 --> 00:05:11.490 and finally, you've got a form that has a button in it. 118 00:05:11.490 --> 00:05:12.860 Don't have to worry too much about that. 119 00:05:12.860 --> 00:05:14.560 We're not gonna be looking into this. 120 00:05:14.560 --> 00:05:17.540 Then you close the div that you started up here. 121 00:05:17.540 --> 00:05:18.860 Finally, you close the article. 122 00:05:18.860 --> 00:05:21.780 You close the list item, which is for the entire thing, 123 00:05:21.780 --> 00:05:23.280 and then, you close your html. 124 00:05:24.130 --> 00:05:26.480 So we sort of understand this code. 125 00:05:26.480 --> 00:05:28.780 Maybe it's not the clearest thing in the world, 126 00:05:28.780 --> 00:05:32.130 but we are going to try to extract some data from it. 127 00:05:32.130 --> 00:05:34.000 So let's do just that. 128 00:05:34.000 --> 00:05:37.450 The first thing is, of course, to create our soup. 129 00:05:37.450 --> 00:05:41.440 So we're gonna do BeautifulSoup of ITEM_HTML 130 00:05:41.440 --> 00:05:43.203 on the html.parser. 131 00:05:44.670 --> 00:05:46.650 Then, we're going to start 132 00:05:46.650 --> 00:05:50.400 by locating the item name, okay? 133 00:05:50.400 --> 00:05:53.410 So the item name, the title of the book, 134 00:05:53.410 --> 00:05:56.200 is, for example, in this a. 135 00:05:56.200 --> 00:05:59.330 You remember, we've got an image there 136 00:05:59.330 --> 00:06:01.040 that has an alt property, 137 00:06:01.040 --> 00:06:03.240 which is the title of the book, 138 00:06:03.240 --> 00:06:06.240 or more normally, we can get it off the actual title. 139 00:06:06.240 --> 00:06:08.110 This is a heading three as you know, 140 00:06:08.110 --> 00:06:09.770 and inside it, there's an a, 141 00:06:09.770 --> 00:06:11.900 and the a has a property, title, 142 00:06:11.900 --> 00:06:13.660 and that's the title of the book. 143 00:06:13.660 --> 00:06:17.560 We cannot get the content out of the string property 144 00:06:17.560 --> 00:06:18.930 because that has a three dots, 145 00:06:18.930 --> 00:06:20.540 and it's missing part of the title, 146 00:06:20.540 --> 00:06:22.133 so we must get it through here. 147 00:06:23.520 --> 00:06:24.950 Okay, let's do that. 148 00:06:24.950 --> 00:06:28.223 Let's create a function which is gonna be find_item_name, 149 00:06:30.697 --> 00:06:31.900 and I'm going to introduce to you 150 00:06:31.900 --> 00:06:33.870 a new way of locating things, 151 00:06:33.870 --> 00:06:38.013 which I think is substantially easier than the default way. 152 00:06:38.940 --> 00:06:43.830 So we're going to identify, from the start of our document, 153 00:06:43.830 --> 00:06:48.750 what path of children will give us our element, 154 00:06:48.750 --> 00:06:49.783 our a element. 155 00:06:50.940 --> 00:06:54.070 So as you can see, our title tag, 156 00:06:54.070 --> 00:06:57.080 our title property here, is in this a tag. 157 00:06:57.080 --> 00:06:59.410 The a tag is in the h3 tag. 158 00:06:59.410 --> 00:07:03.170 The h3 tag is in the article tag. 159 00:07:03.170 --> 00:07:05.400 The article tag is in the li tag, 160 00:07:05.400 --> 00:07:08.143 which is in the body tag, which is in the html tag. 161 00:07:09.990 --> 00:07:13.100 So we're going to look at 162 00:07:14.290 --> 00:07:15.570 any of these parents. 163 00:07:15.570 --> 00:07:16.640 We can start at anywhere. 164 00:07:16.640 --> 00:07:18.480 We can start in the li. 165 00:07:18.480 --> 00:07:19.890 We can start at the article. 166 00:07:19.890 --> 00:07:21.550 We can start at the h3, 167 00:07:21.550 --> 00:07:25.623 and we're gonna say article is the first one, for example. 168 00:07:26.490 --> 00:07:27.623 Then comes h3. 169 00:07:28.760 --> 00:07:29.863 Then comes a, 170 00:07:31.100 --> 00:07:32.490 and you can see that, 171 00:07:32.490 --> 00:07:35.320 you can see that because this div here, for example, 172 00:07:35.320 --> 00:07:38.493 is in the same level as this p tag. 173 00:07:39.490 --> 00:07:41.610 The indentation is not very good, 174 00:07:41.610 --> 00:07:44.650 but you can see that the div tag opens here 175 00:07:44.650 --> 00:07:46.200 and it closes here, 176 00:07:46.200 --> 00:07:49.290 which means the p tag is not inside it, 177 00:07:49.290 --> 00:07:50.380 and the p tag closes here, 178 00:07:50.380 --> 00:07:52.930 which means the h3 tag is not inside it. 179 00:07:52.930 --> 00:07:54.320 So these are all on the same level, 180 00:07:54.320 --> 00:07:57.370 which means they're all children of the article, 181 00:07:57.370 --> 00:07:58.930 and the a is a children of an h3. 182 00:07:58.930 --> 00:08:01.510 So what we've got essentially in our locator 183 00:08:02.470 --> 00:08:04.200 is we've got an article. 184 00:08:04.200 --> 00:08:06.010 Then separated by a space, we have our h3, 185 00:08:06.010 --> 00:08:07.160 and then, we have an a, 186 00:08:08.350 --> 00:08:11.660 and this sort of hierarchy allows BeautifulSoup 187 00:08:11.660 --> 00:08:15.493 to find elements very easily, okay? 188 00:08:16.780 --> 00:08:20.700 If you wanted to narrow down into a specific article, 189 00:08:20.700 --> 00:08:22.863 for example, if you've had many articles, 190 00:08:23.860 --> 00:08:27.183 notice how this article has a class of product_pod. 191 00:08:28.660 --> 00:08:29.770 So you can actually say 192 00:08:29.770 --> 00:08:31.633 article.product_pod, 193 00:08:35.350 --> 00:08:36.550 and what this means now 194 00:08:36.550 --> 00:08:38.020 is you're looking for the article 195 00:08:38.020 --> 00:08:40.970 that has class product_pod, 196 00:08:40.970 --> 00:08:43.013 and inside it, you're looking for an h3, 197 00:08:43.013 --> 00:08:45.680 and inside it, you're looking for an a, 198 00:08:45.680 --> 00:08:47.810 and this, once you do, for example, 199 00:08:47.810 --> 00:08:50.777 item_name is soup.select_one 200 00:08:52.810 --> 00:08:54.460 locator 201 00:08:54.460 --> 00:08:58.163 is going to give you whatever it is that this results in. 202 00:08:59.010 --> 00:09:02.620 This thing here is known as a CSS locator 203 00:09:02.620 --> 00:09:03.850 and is really powerful. 204 00:09:03.850 --> 00:09:06.890 It allows us to essentially do this sort of thing, 205 00:09:06.890 --> 00:09:10.690 and it's the way that CSS, the CSS language, 206 00:09:10.690 --> 00:09:11.780 also locates things. 207 00:09:11.780 --> 00:09:14.270 We're not gonna look into CSS in this course, 208 00:09:14.270 --> 00:09:15.410 or at least not yet, 209 00:09:15.410 --> 00:09:18.163 but that's what it's called, essentially. 210 00:09:19.630 --> 00:09:21.880 In its simplest terms, we can get the parent, 211 00:09:22.750 --> 00:09:25.250 narrow it down by a class if we want, 212 00:09:25.250 --> 00:09:26.750 and then go down to a child, 213 00:09:26.750 --> 00:09:28.030 and then another child and so forth 214 00:09:28.030 --> 00:09:31.880 until we have a hierarchy that is not repeated 215 00:09:31.880 --> 00:09:33.330 throughout the site. 216 00:09:33.330 --> 00:09:36.527 So if you only have one article with a product_pod, 217 00:09:36.527 --> 00:09:37.920 one h3, and one a, 218 00:09:37.920 --> 00:09:40.330 then that's gonna give you this hierarchy here 219 00:09:40.330 --> 00:09:41.530 that you can search for. 220 00:09:43.220 --> 00:09:44.320 Now that you've got the, 221 00:09:44.320 --> 00:09:45.410 we're not gonna call it item_name. 222 00:09:45.410 --> 00:09:46.910 We're gonna call it item_link. 223 00:09:46.910 --> 00:09:48.070 Now you've got the item_link. 224 00:09:48.070 --> 00:09:49.617 We can say item_name 225 00:09:49.617 --> 00:09:53.193 is item_link.attrs title. 226 00:09:54.870 --> 00:09:58.330 Again, this accesses the attributes of the link. 227 00:09:58.330 --> 00:09:59.723 Our link is here, 228 00:10:01.540 --> 00:10:04.390 and we're gonna access the title property 229 00:10:04.390 --> 00:10:06.293 or attribute of that link, 230 00:10:07.410 --> 00:10:09.460 and then, we can print the item_name out. 231 00:10:11.820 --> 00:10:12.790 Let's run the function, 232 00:10:12.790 --> 00:10:15.513 and remember to run your middle_html file, 233 00:10:16.350 --> 00:10:17.330 and you can see that we get 234 00:10:17.330 --> 00:10:19.123 A Light in the Attic printed out. 235 00:10:21.800 --> 00:10:23.360 Now using these locators, 236 00:10:23.360 --> 00:10:25.810 you can essentially locate anything, 237 00:10:25.810 --> 00:10:27.960 so here's a small challenge for you. 238 00:10:27.960 --> 00:10:30.940 Using a locator, you're gonna actually use the same one, 239 00:10:30.940 --> 00:10:35.400 find out the link to this item's page. 240 00:10:35.400 --> 00:10:39.100 So what you're looking for is this href here. 241 00:10:39.100 --> 00:10:40.250 I'm sure you can do it. 242 00:10:41.600 --> 00:10:42.630 I hope you managed. 243 00:10:42.630 --> 00:10:46.190 All that we have to do is essentially access the same, 244 00:10:46.190 --> 00:10:47.530 the same locator, 245 00:10:47.530 --> 00:10:49.410 but instead of the title, 246 00:10:49.410 --> 00:10:53.040 access another attribute which is the href attribute. 247 00:10:53.040 --> 00:10:54.190 So we're going to copy. 248 00:10:57.190 --> 00:11:00.540 Paste it in here, and we're gonna call it find_item_link, 249 00:11:00.540 --> 00:11:03.820 and then, we're going to access the href attribute here. 250 00:11:03.820 --> 00:11:05.130 Again, it's the same element, 251 00:11:05.130 --> 00:11:07.210 article with product_pod class, 252 00:11:07.210 --> 00:11:09.460 the h3 tag, which is the child, 253 00:11:09.460 --> 00:11:11.503 and the a tag, which is a child of that, 254 00:11:12.734 --> 00:11:14.800 and then, we're gonna select that, 255 00:11:14.800 --> 00:11:18.147 and finally, we're gonna access the attribute of that, 256 00:11:18.147 --> 00:11:19.470 and we're gonna print it out, 257 00:11:19.470 --> 00:11:21.801 but of course, now that we have this item_link, 258 00:11:21.801 --> 00:11:23.770 that's a pretty good name for our link. 259 00:11:23.770 --> 00:11:26.050 We're gonna just do this, 260 00:11:26.050 --> 00:11:28.330 so access the item, 261 00:11:28.330 --> 00:11:30.090 eh, sorry, the tag, 262 00:11:30.090 --> 00:11:32.230 and then access the href property of that, 263 00:11:32.230 --> 00:11:33.743 and then, we can print that out. 264 00:11:36.670 --> 00:11:38.190 Let's delete this 265 00:11:40.700 --> 00:11:41.763 and run again, 266 00:11:43.440 --> 00:11:45.260 and now, you can see that we get the title, 267 00:11:45.260 --> 00:11:47.163 and then we get the link here as well. 268 00:11:48.140 --> 00:11:50.790 This is a relative link, 269 00:11:50.790 --> 00:11:53.960 so it doesn't give you the entire domain name, 270 00:11:53.960 --> 00:11:55.790 and, you know, the website and everything. 271 00:11:55.790 --> 00:11:56.623 That's okay. 272 00:11:56.623 --> 00:11:58.920 I mean, we'll learn how to deal with that later on. 273 00:12:00.810 --> 00:12:05.023 Okay, using this locator, now we can find other things. 274 00:12:05.880 --> 00:12:07.827 So we can find, for example, the item price 275 00:12:07.827 --> 00:12:10.022 and the item rating. 276 00:12:10.022 --> 00:12:12.970 It's not so easy to do 277 00:12:12.970 --> 00:12:15.820 because you're gonna have to do some other calculations. 278 00:12:15.820 --> 00:12:19.210 For example, if you wanted to find the item's price, 279 00:12:19.210 --> 00:12:20.810 which is here, 280 00:12:20.810 --> 00:12:23.010 you're gonna look at article. 281 00:12:23.010 --> 00:12:24.600 Then you're gonna potentially look at a p 282 00:12:24.600 --> 00:12:27.170 that has price_color class, 283 00:12:27.170 --> 00:12:28.460 and then, you're gonna look at its contents, 284 00:12:28.460 --> 00:12:29.823 the string contents of it, 285 00:12:30.760 --> 00:12:32.500 but this is not a number. 286 00:12:32.500 --> 00:12:33.840 It is a string, 287 00:12:33.840 --> 00:12:36.660 so you may want to strip out the pound sign 288 00:12:36.660 --> 00:12:40.163 and use a regular expression to extract this group. 289 00:12:41.160 --> 00:12:43.140 I think this would be a great exercise 290 00:12:43.140 --> 00:12:46.180 for us to review what we've learned in the last section 291 00:12:46.180 --> 00:12:47.270 and give it a go. 292 00:12:47.270 --> 00:12:51.290 Try to extract 51.77 as a float 293 00:12:52.140 --> 00:12:55.770 using BeautifulSoup and the regular expressions. 294 00:12:55.770 --> 00:12:56.603 Give it a go. 295 00:12:56.603 --> 00:12:57.923 Pause the video and go for it. 296 00:12:58.910 --> 00:12:59.743 I hope you managed. 297 00:12:59.743 --> 00:13:03.040 It's a bit of a more complicated way to go about things, 298 00:13:03.040 --> 00:13:05.873 but what we're going to do is, first of all, locate this. 299 00:13:05.873 --> 00:13:08.290 First of all, get this string out with BeautifulSoup. 300 00:13:08.290 --> 00:13:10.433 So I'm going to create a new function, 301 00:13:12.174 --> 00:13:13.507 find_item_price, 302 00:13:14.860 --> 00:13:17.180 and then, I'm going to define my locator, 303 00:13:17.180 --> 00:13:20.290 which is gonna be article.product_pod, 304 00:13:20.290 --> 00:13:22.240 and inside it, we've got p.price_color. 305 00:13:23.120 --> 00:13:26.020 So here, we're accessing the article that has this class 306 00:13:26.020 --> 00:13:27.457 and the p that has this class, 307 00:13:27.457 --> 00:13:30.520 and that's gonna give us a pretty good understanding, 308 00:13:30.520 --> 00:13:32.250 or it's gonna give BeautifulSoup a good understanding 309 00:13:32.250 --> 00:13:33.543 of how to get there. 310 00:13:34.910 --> 00:13:37.060 Then, we're going to access the item_price, 311 00:13:38.060 --> 00:13:42.963 which is going to be soup.select_one locator .string. 312 00:13:43.840 --> 00:13:46.810 All that's happening here is we're finding the tag, 313 00:13:46.810 --> 00:13:48.760 and we're accessing the string content of it. 314 00:13:48.760 --> 00:13:52.450 So this is gonna give us pound sign 51.77 315 00:13:52.450 --> 00:13:53.370 or however much it was. 316 00:13:53.370 --> 00:13:54.760 Yeah, 51.77. 317 00:13:54.760 --> 00:13:56.060 So this is giving us that. 318 00:13:58.110 --> 00:13:59.490 Now, we have this. 319 00:13:59.490 --> 00:14:03.080 We have to use a regular expression in order to extract it. 320 00:14:03.080 --> 00:14:04.310 So go into the top of the file. 321 00:14:04.310 --> 00:14:07.430 I'm going to import the RE module, 322 00:14:07.430 --> 00:14:08.263 import re. 323 00:14:09.200 --> 00:14:11.420 Notice how I am doing this import re 324 00:14:11.420 --> 00:14:13.070 at the top of the file 325 00:14:13.070 --> 00:14:16.200 and from blah blah blah import something else 326 00:14:16.200 --> 00:14:17.210 directly underneath it. 327 00:14:17.210 --> 00:14:19.230 This is Python sort of convention 328 00:14:19.230 --> 00:14:22.150 to do these type of imports at the top, 329 00:14:22.150 --> 00:14:24.500 these type of imports just directly below them. 330 00:14:25.770 --> 00:14:27.620 Now that we've got the re module, 331 00:14:27.620 --> 00:14:29.020 we can define our pattern, 332 00:14:29.020 --> 00:14:31.070 the regular expression we're looking for, 333 00:14:31.910 --> 00:14:34.813 and this is going to be, first of all, the pound symbol, 334 00:14:35.920 --> 00:14:37.196 51.77. 335 00:14:37.196 --> 00:14:38.800 This is going to match, 336 00:14:38.800 --> 00:14:43.800 and we want to extract 51.77 as a number. 337 00:14:44.120 --> 00:14:46.030 So we're gonna put it inside these brackets 338 00:14:46.030 --> 00:14:48.183 so that we can extract it using the group. 339 00:14:49.570 --> 00:14:52.590 Then of course, we're not interested in this 51.77 only, 340 00:14:52.590 --> 00:14:54.460 but if the price were to change, 341 00:14:54.460 --> 00:14:56.860 we'd wanna get the changed price, 342 00:14:56.860 --> 00:14:59.620 so instead of putting 51.77 in here, 343 00:14:59.620 --> 00:15:02.130 we're gonna have to use something that will allow us 344 00:15:02.130 --> 00:15:05.230 to get a range of numbers, 345 00:15:05.230 --> 00:15:06.410 and the way we're gonna do that 346 00:15:06.410 --> 00:15:09.670 is by doing zero to nine inside the square brackets. 347 00:15:09.670 --> 00:15:12.750 This is a range of numbers from zero to nine. 348 00:15:12.750 --> 00:15:14.150 Then we're gonna do plus, 349 00:15:14.150 --> 00:15:16.800 so that's gonna give us one or more of these numbers, 350 00:15:17.850 --> 00:15:21.530 and then, we're going to make sure to put a backslash 351 00:15:21.530 --> 00:15:22.990 in front of the dot 352 00:15:22.990 --> 00:15:26.170 so that it matches the dot and not anything, 353 00:15:26.170 --> 00:15:28.250 which is what it means in regular expressions, 354 00:15:28.250 --> 00:15:30.580 and for the 77, we're going to do the same, 355 00:15:30.580 --> 00:15:32.853 zero to nine plus. 356 00:15:34.990 --> 00:15:37.460 Then we're going to search and get our groups. 357 00:15:37.460 --> 00:15:40.620 So our groups or our matcher 358 00:15:40.620 --> 00:15:44.163 is going to be re.search pattern and item_price, 359 00:15:45.970 --> 00:15:49.970 and again, this is going to give us two groups, 360 00:15:49.970 --> 00:15:51.990 group zero, which is going to be 361 00:15:51.990 --> 00:15:54.173 the entire thing that matches, 362 00:15:55.240 --> 00:15:57.660 and then, matcher.group one, 363 00:15:57.660 --> 00:15:59.470 which is going to be the first group 364 00:15:59.470 --> 00:16:01.090 that is between the brackets. 365 00:16:01.090 --> 00:16:05.710 So that's gonna be 51.77 without the pound symbol. 366 00:16:05.710 --> 00:16:08.780 Okay, this is gonna be 51.77. 367 00:16:08.780 --> 00:16:11.750 This is gonna be 51.77 without the symbol. 368 00:16:11.750 --> 00:16:14.693 So we can indeed turn it into a float. 369 00:16:16.440 --> 00:16:20.920 Okay, we're printing out the float of this matcher, 370 00:16:20.920 --> 00:16:23.153 and so, let's run that and see what happens, 371 00:16:25.660 --> 00:16:28.870 and as you can see, we get 51.77 with the pound sign 372 00:16:28.870 --> 00:16:33.730 and 51.77 without the pound sign, okay? 373 00:16:33.730 --> 00:16:35.680 Let's say we wanted to print it out but 374 00:16:37.160 --> 00:16:39.640 at a 20% discount. 375 00:16:39.640 --> 00:16:41.010 We could multiply by 0.8, 376 00:16:41.010 --> 00:16:43.120 and this should work because it's a number now. 377 00:16:43.120 --> 00:16:44.167 It's not a string, 378 00:16:45.743 --> 00:16:46.860 and as you can see, it works. 379 00:16:46.860 --> 00:16:48.853 We get 41.416. 380 00:16:49.920 --> 00:16:51.680 So that's pretty awesome. 381 00:16:51.680 --> 00:16:55.040 We have now learned how to extract a number 382 00:16:55.040 --> 00:16:59.240 using BeautifulSoup and regular expressions from a website. 383 00:16:59.240 --> 00:17:01.360 So for example, the things we can do now, 384 00:17:01.360 --> 00:17:03.310 once we learn, of course, how to interact with the website, 385 00:17:03.310 --> 00:17:05.920 which we will do shortly in this section, 386 00:17:05.920 --> 00:17:09.230 you can, for example, track item prices online. 387 00:17:09.230 --> 00:17:12.220 For example, let's say you played badminton 388 00:17:12.220 --> 00:17:14.860 and you wanna track the price of badminton rackets 389 00:17:14.860 --> 00:17:17.380 over time in order to find out when you should buy one. 390 00:17:17.380 --> 00:17:19.550 You could do this with a simple Python app 391 00:17:19.550 --> 00:17:21.300 that essentially downloads the site 392 00:17:21.300 --> 00:17:24.243 and looks through it using something like this. 393 00:17:26.100 --> 00:17:28.730 Finally, let's find out the item's rating. 394 00:17:28.730 --> 00:17:32.150 The rating is gonna be a bit more complicated 395 00:17:32.150 --> 00:17:37.150 because we have to look at exactly one attribute, 396 00:17:37.200 --> 00:17:39.500 which is this Three here, 397 00:17:39.500 --> 00:17:41.740 but of course, this is in a p tag, 398 00:17:41.740 --> 00:17:44.150 and there are many p tags around, 399 00:17:44.150 --> 00:17:45.913 so let's find this out, 400 00:17:48.750 --> 00:17:49.960 and let's go down here. 401 00:17:49.960 --> 00:17:52.410 We're gonna do def find_item_rating. 402 00:17:53.960 --> 00:17:57.203 The locator is going to be article.product_pod, 403 00:17:58.540 --> 00:18:00.080 product_pod, 404 00:18:00.080 --> 00:18:02.973 and inside it, p.star-rating. 405 00:18:04.340 --> 00:18:06.120 That is because the paragraph we're looking for 406 00:18:06.120 --> 00:18:09.580 is the one that has this class, star-rating. 407 00:18:09.580 --> 00:18:13.480 In CSS, classes are always separated by spaces, 408 00:18:13.480 --> 00:18:16.010 so what this p tag has is two classes, 409 00:18:16.010 --> 00:18:19.030 star-rating and Three. 410 00:18:19.030 --> 00:18:20.053 So we're looking for the paragraph 411 00:18:20.053 --> 00:18:22.070 that has the star-rating class, 412 00:18:22.070 --> 00:18:23.820 and it also has other classes, 413 00:18:23.820 --> 00:18:26.650 but we're not limited to only this one. 414 00:18:26.650 --> 00:18:29.530 So this locator will find that paragraph. 415 00:18:29.530 --> 00:18:31.510 It has a p.star-rating, 416 00:18:31.510 --> 00:18:33.770 and if we wanted to also find it with the Three class, 417 00:18:33.770 --> 00:18:35.440 we could do this, 418 00:18:35.440 --> 00:18:37.440 but we don't want to find it with two classes 419 00:18:37.440 --> 00:18:39.230 just in case the rating changes, 420 00:18:39.230 --> 00:18:41.343 so we're gonna stick it with this only. 421 00:18:42.960 --> 00:18:45.800 Okay, the star_rating_element, 422 00:18:45.800 --> 00:18:47.560 the tag or the star_rating_tag 423 00:18:47.560 --> 00:18:52.560 is going to be soup.select_one, here, locator. 424 00:18:52.710 --> 00:18:55.993 So this gives us the tag, okay? 425 00:18:56.910 --> 00:19:00.360 Now, we can get the class tags out of it. 426 00:19:00.360 --> 00:19:02.270 So the classes are 427 00:19:02.270 --> 00:19:05.993 star_rating_tag.attrs class, 428 00:19:07.021 --> 00:19:09.520 and this is now, when BeautifulSoup extract this, 429 00:19:09.520 --> 00:19:12.183 is going to be two classes, 430 00:19:13.080 --> 00:19:14.370 these two. 431 00:19:14.370 --> 00:19:15.890 This is what it's gonna give us, 432 00:19:15.890 --> 00:19:19.230 so we need to find out what this is, 433 00:19:19.230 --> 00:19:22.150 but remember, it could be the first one. 434 00:19:22.150 --> 00:19:24.670 It doesn't necessarily have to be the second class, 435 00:19:24.670 --> 00:19:27.730 so this list could be like this. 436 00:19:27.730 --> 00:19:31.710 We cannot just select the second element. 437 00:19:31.710 --> 00:19:33.520 That would be too easy. 438 00:19:33.520 --> 00:19:34.750 So what we're going to do is 439 00:19:34.750 --> 00:19:37.900 we're going to recreate this classes list 440 00:19:37.900 --> 00:19:40.440 but without this star-rating. 441 00:19:40.440 --> 00:19:43.220 I'm sure you can do that using either a list comprehension 442 00:19:43.220 --> 00:19:44.920 or the filter function. 443 00:19:44.920 --> 00:19:45.753 Give it a go. 444 00:19:46.700 --> 00:19:47.850 I'm sure you managed. 445 00:19:47.850 --> 00:19:49.110 There are two ways that we could do this. 446 00:19:49.110 --> 00:19:52.170 We could say rating_classes is 447 00:19:52.170 --> 00:19:54.700 r for r in classes 448 00:19:54.700 --> 00:19:58.363 if r is not equal to star-rating. 449 00:19:59.900 --> 00:20:00.790 This would be one way, 450 00:20:00.790 --> 00:20:03.270 or we could do rating_classes equals filter 451 00:20:03.270 --> 00:20:06.360 of lambda x for x is not 452 00:20:07.200 --> 00:20:10.423 star-rating in classes. 453 00:20:11.680 --> 00:20:14.800 Okay, either of these would be perfectly suitable. 454 00:20:14.800 --> 00:20:18.490 The code that you'll see online has this. 455 00:20:18.490 --> 00:20:21.830 So we're gonna leave the list comprehension in the video. 456 00:20:21.830 --> 00:20:25.040 If you wanna look out at the other way of doing things, 457 00:20:25.040 --> 00:20:26.680 check out the video which is linked 458 00:20:26.680 --> 00:20:28.430 at the start of the lecture 459 00:20:28.430 --> 00:20:30.307 and also at the start of this section 460 00:20:30.307 --> 00:20:31.507 and also in the lecture. 461 00:20:33.040 --> 00:20:36.650 Now this list here should contain a single element, 462 00:20:36.650 --> 00:20:37.543 this Three. 463 00:20:38.500 --> 00:20:42.963 So we can just say print rating_classes zero. 464 00:20:43.930 --> 00:20:45.800 We're assuming that the paragraph 465 00:20:45.800 --> 00:20:48.220 is not going to have any other tags, 466 00:20:48.220 --> 00:20:52.120 say for example, a hello class. 467 00:20:52.120 --> 00:20:54.450 Would be weird, but you know, it can happen, 468 00:20:54.450 --> 00:20:59.450 and this method, if that's the case, is not ideal for that, 469 00:20:59.870 --> 00:21:03.260 but through looking at a few of the items on the website, 470 00:21:03.260 --> 00:21:05.550 it seems that there are only ever two classes, 471 00:21:05.550 --> 00:21:08.370 star-rating and the rating. 472 00:21:08.370 --> 00:21:10.983 So that could be Four or Five or Three and so on. 473 00:21:12.640 --> 00:21:17.283 Finally, let's call this one find_item_rating and run this, 474 00:21:18.550 --> 00:21:20.010 and there you have it down here. 475 00:21:20.010 --> 00:21:22.200 We've got the number of stars in this book. 476 00:21:22.200 --> 00:21:24.600 So we could use this now to search the books, 477 00:21:24.600 --> 00:21:27.730 so we could, for example, download the entire page 478 00:21:27.730 --> 00:21:30.040 and find which books have five stars 479 00:21:30.040 --> 00:21:31.553 and are under 20 pounds, 480 00:21:32.400 --> 00:21:35.250 and that would be something that we could do, 481 00:21:35.250 --> 00:21:36.650 but before we do that, of course, 482 00:21:36.650 --> 00:21:41.050 we have to learn to extract the content from the site 483 00:21:41.050 --> 00:21:43.150 by downloading the website. 484 00:21:43.150 --> 00:21:44.880 So we're going to do that very soon, 485 00:21:44.880 --> 00:21:46.830 but first, I just wanted to tell you a little bit 486 00:21:46.830 --> 00:21:50.330 about how to improve on this structure. 487 00:21:50.330 --> 00:21:51.490 You're not always going to, 488 00:21:51.490 --> 00:21:53.833 or you're not gonna want to have, 489 00:21:54.820 --> 00:21:56.270 you know, these top-level functions 490 00:21:56.270 --> 00:21:57.830 for each thing you're finding. 491 00:21:57.830 --> 00:21:58.920 You're not gonna want your functions 492 00:21:58.920 --> 00:22:00.800 to be printing things out necessarily, 493 00:22:00.800 --> 00:22:02.530 so we're gonna make things a bit better. 494 00:22:02.530 --> 00:22:05.670 We're going to start separating things into classes 495 00:22:05.670 --> 00:22:08.070 and things like that just so you have a better idea 496 00:22:08.070 --> 00:22:12.520 of how I'd recommend that you go about parsing the HTML 497 00:22:12.520 --> 00:22:15.570 and sort of dealing with it nicely in your programmes. 498 00:22:15.570 --> 00:22:18.070 So with that said, I'll see you on the next video.