1 1 00:00:01,280 --> 00:00:02,540 So what we're doing here is 2 2 00:00:02,540 --> 00:00:04,580 we're going to look at the website 3 3 00:00:04,580 --> 00:00:06,990 that we loaded in the previous video just so we can 4 4 00:00:06,990 --> 00:00:10,790 figure out how to find the price of the element. 5 5 00:00:10,790 --> 00:00:13,250 In this case I'm looking at an Ipad Pro 6 6 00:00:13,250 --> 00:00:17,750 but any item in this website will be the same. 7 7 00:00:17,750 --> 00:00:19,660 So that's something to think about. 8 8 00:00:19,660 --> 00:00:22,690 Whenever you look at an item in this department store 9 9 00:00:22,690 --> 00:00:23,940 you can see that the structure 10 10 00:00:23,940 --> 00:00:26,870 of the page is pretty much identical. 11 11 00:00:26,870 --> 00:00:28,840 And that's just because these pages 12 12 00:00:28,840 --> 00:00:31,140 are also generated programmatically 13 13 00:00:31,140 --> 00:00:33,340 so it makes sense to keep them all the same. 14 14 00:00:34,360 --> 00:00:37,500 When it comes to retrieving the data 15 15 00:00:37,500 --> 00:00:41,300 we need to make sure that the price is available 16 16 00:00:41,300 --> 00:00:43,930 when you first request the page. 17 17 00:00:43,930 --> 00:00:45,830 So when you first load the page. 18 18 00:00:45,830 --> 00:00:47,070 The reason I'm saying this is because 19 19 00:00:47,070 --> 00:00:50,260 there are some websites where the page loads 20 20 00:00:50,260 --> 00:00:52,110 and after the page has loaded, 21 21 00:00:52,110 --> 00:00:55,170 they then load the price dynamically, 22 22 00:00:55,170 --> 00:00:57,920 using the programming language called JavaScript. 23 23 00:00:57,920 --> 00:00:59,570 JavaScript is a programming language 24 24 00:00:59,570 --> 00:01:03,490 that runs on your browser, so once you load the page 25 25 00:01:03,490 --> 00:01:07,520 JavaScript then runs and can modify the page. 26 26 00:01:07,520 --> 00:01:10,240 If that happens, the price may not be available 27 27 00:01:10,240 --> 00:01:11,870 when you first load the page, 28 28 00:01:11,870 --> 00:01:13,800 unless you run the JavaScript first. 29 29 00:01:13,800 --> 00:01:17,500 Our code is not going to work for websites that do that. 30 30 00:01:17,500 --> 00:01:22,160 Just because the requests module is not able to evaluate and 31 31 00:01:22,160 --> 00:01:26,010 execute the JavaScript code that would make that change. 32 32 00:01:26,010 --> 00:01:28,640 Fortunately for us, johnlewis.com does not use 33 33 00:01:28,640 --> 00:01:31,750 JavaScript to modify the page contents, so it is fine. 34 34 00:01:31,750 --> 00:01:35,190 But many other websites do, and for that you need a 35 35 00:01:35,190 --> 00:01:38,860 slightly more complicated process in order to achieve that. 36 36 00:01:38,860 --> 00:01:42,230 In this case we're interested in the price that we see here. 37 37 00:01:42,230 --> 00:01:43,410 So what we're going to do is 38 38 00:01:43,410 --> 00:01:46,560 we're going to right-click it and inspect it. 39 39 00:01:46,560 --> 00:01:49,400 Inspecting something in Google Chrome 40 40 00:01:49,400 --> 00:01:52,520 is going to bring up this stuff here. 41 41 00:01:52,520 --> 00:01:54,360 This thing that you see, 42 42 00:01:54,360 --> 00:01:56,950 is the html code that makes up the page. 43 43 00:01:56,950 --> 00:01:58,370 This is what we received when we 44 44 00:01:58,370 --> 00:01:59,963 requested the page earlier on. 45 45 00:02:03,830 --> 00:02:08,600 If we have this code here we can understand it and we can 46 46 00:02:08,600 --> 00:02:12,360 potentially go and retrieve something from within it. 47 47 00:02:12,360 --> 00:02:14,780 We're going to use BeautifulSoup to do that. 48 48 00:02:14,780 --> 00:02:17,070 Well, the first thing is to make sure that we understand 49 49 00:02:17,070 --> 00:02:21,900 as the programmer where the price is located within the page 50 50 00:02:21,900 --> 00:02:25,420 relative to other elements that are in the page. 51 51 00:02:25,420 --> 00:02:26,640 This can be a bit tricky. 52 52 00:02:26,640 --> 00:02:28,630 What we're interested in is finding 53 53 00:02:28,630 --> 00:02:31,290 the element that contains the price. 54 54 00:02:31,290 --> 00:02:33,680 So in the case of johnlewis.com 55 55 00:02:33,680 --> 00:02:37,520 you can see that when we hover over this element here 56 56 00:02:37,520 --> 00:02:40,200 that says p class equals 57 57 00:02:40,200 --> 00:02:42,700 price price dash dash large 58 58 00:02:42,700 --> 00:02:44,830 that seems to contain the price. 59 59 00:02:44,830 --> 00:02:47,280 You can see that with the blue background 60 60 00:02:47,280 --> 00:02:50,930 that appears behind the element when we hover over it. 61 61 00:02:50,930 --> 00:02:52,840 This means that this paragraph, 62 62 00:02:52,840 --> 00:02:56,100 that's what the p stands for, contains the price. 63 63 00:02:56,100 --> 00:02:58,530 And you can see that if you open it up 64 64 00:02:58,530 --> 00:03:02,340 the price is indeed there, you can see that the pound sign 65 65 00:03:02,340 --> 00:03:05,900 and then one comma four six nine point zero zero. 66 66 00:03:05,900 --> 00:03:08,300 And you can also see that there are some quotation marks 67 67 00:03:08,300 --> 00:03:10,640 and there's also a lot of white space 68 68 00:03:10,640 --> 00:03:14,110 because this quotation mark is quite far away from the text. 69 69 00:03:14,110 --> 00:03:16,650 All of these things are surmountable problems, 70 70 00:03:16,650 --> 00:03:19,163 so we can deal with those in code. 71 71 00:03:20,010 --> 00:03:21,700 As long as we can tell our code 72 72 00:03:21,700 --> 00:03:26,160 how to find this string here, 1469.00 73 73 00:03:26,160 --> 00:03:29,340 we will be able to then say this Ipad is 74 74 00:03:29,340 --> 00:03:31,720 almost one and a half thousand pounds. 75 75 00:03:31,720 --> 00:03:35,080 So what we have to do is to find a way to tell BeautifulSoup 76 76 00:03:35,080 --> 00:03:38,730 how to find this text here so that we can understand. 77 77 00:03:38,730 --> 00:03:39,670 And the way that we're gonna do that 78 78 00:03:39,670 --> 00:03:42,250 is by using the attributes of this paragraph. 79 79 00:03:42,250 --> 00:03:45,510 We're gonna say that the price is inside 80 80 00:03:45,510 --> 00:03:50,150 a p-tag, a paragraph tag, and it has the class 81 81 00:03:50,150 --> 00:03:53,910 of price price dash dash large. 82 82 00:03:53,910 --> 00:03:57,190 With that information, paragraph tag and the class 83 83 00:03:57,190 --> 00:03:58,630 we're going to be able to tell 84 84 00:03:58,630 --> 00:04:00,970 BeautifulSoup how to find the price. 85 85 00:04:00,970 --> 00:04:03,690 If you're using a different website that is not John Lewis 86 86 00:04:03,690 --> 00:04:05,540 then you're gonna have to do this on your own. 87 87 00:04:05,540 --> 00:04:06,890 You're gonna have to find 88 88 00:04:06,890 --> 00:04:09,500 the element that contains the price and 89 89 00:04:09,500 --> 00:04:14,270 figure out how you can identify that element in the page. 90 90 00:04:14,270 --> 00:04:17,090 So essentially you're gonna try to see 91 91 00:04:17,090 --> 00:04:21,510 what element contains the price and what attribute it's got. 92 92 00:04:21,510 --> 00:04:23,900 Once you know that, we can go back to the code 93 93 00:04:23,900 --> 00:04:26,930 and tell BeautifulSoup how to do this. 94 94 00:04:26,930 --> 00:04:27,763 Let's do it.