1 00:00:00,600 --> 00:00:07,560 Welcome to listen 40 which roughly covers pages 240 to 248 of the automate the boring stuff with Python 2 00:00:07,560 --> 00:00:12,650 textbook in this lesson we'll learn how to write programs that pull information off of web pages. 3 00:00:12,660 --> 00:00:14,650 This is known as Web scraping. 4 00:00:14,660 --> 00:00:20,400 It will help if you know H-G email and CSSA selectors already there are links to tutorials for these 5 00:00:20,400 --> 00:00:21,140 in the course notes. 6 00:00:21,150 --> 00:00:25,740 But there is also a way that you can get the browser to figure out the CSA selector for you and we'll 7 00:00:25,740 --> 00:00:27,800 go over that in this lesson. 8 00:00:28,110 --> 00:00:33,960 When your browser downloads a web page it's downloading a plain text file formatted as h t m l h d m 9 00:00:33,960 --> 00:00:35,240 l sounds a bit cryptic. 10 00:00:35,250 --> 00:00:38,070 It stands for HyperText Markup Language. 11 00:00:38,280 --> 00:00:40,410 Yeah I'm guessing that did clarify things. 12 00:00:40,410 --> 00:00:46,200 Basically T.M. is the text that tells the browser how to make the web page look you can see the H.T. 13 00:00:46,200 --> 00:00:51,660 mail for any web page by pressing control you in your browser or right clicking the page and selecting 14 00:00:51,870 --> 00:00:55,200 view source. 15 00:00:55,260 --> 00:01:03,930 H.T. email is mostly text with these angled bracket bracketed things called each DML elements bring 16 00:01:03,930 --> 00:01:11,400 up the developer tools in your web browser on Chrome and Firefox is done by pressing F 12. 17 00:01:11,400 --> 00:01:17,420 You can hover over the H.T. M-L in the elements pain and it will point out what part of the page this 18 00:01:17,420 --> 00:01:20,080 aged female is responsible for. 19 00:01:20,940 --> 00:01:29,580 Alternatively you can right click on a part of the page and select inspect element and then will bring 20 00:01:29,580 --> 00:01:34,230 up the HTL male element for that piece of the page that you clicked on 21 00:01:34,230 --> 00:01:40,530 . 22 00:01:40,620 --> 00:01:44,540 And this will be useful when we need to pull information off of our Web site. 23 00:01:44,550 --> 00:01:49,500 Now the request module handles downloading the web page itself but that just gets us this huge string 24 00:01:49,500 --> 00:01:52,220 of text of the pages. 25 00:01:52,230 --> 00:01:53,190 H t m l. 26 00:01:53,190 --> 00:01:56,600 In order to locate the text we need inside this huge string. 27 00:01:56,670 --> 00:02:02,340 We need to parse the HD and now the third party module beautiful suit makes this easy and beautiful 28 00:02:02,340 --> 00:02:08,580 soup is a third party module so you have to install it by running Pip install beautiful soup for to 29 00:02:08,580 --> 00:02:12,760 see if the install worked type import B-S for. 30 00:02:12,990 --> 00:02:17,490 And if there are no error messages then the module has been installed successfully. 31 00:02:17,510 --> 00:02:23,040 Notice that even though the name of the module is beautiful soup the module the name that used to import 32 00:02:23,040 --> 00:02:28,240 it is B-S for instance for the fourth version of beautiful soup. 33 00:02:28,320 --> 00:02:34,960 So let's work through an example of web scraping the price information off of an Amazon page. 34 00:02:36,420 --> 00:02:42,930 So we want our Python script to download this web page and then find this price information. 35 00:02:43,080 --> 00:02:50,730 So I'm going to first copy this you r l and we've imported the beautiful soup module but we'll also 36 00:02:50,730 --> 00:02:56,790 need to import requests which we covered in the last lesson and this will handle the actual downloading 37 00:03:02,270 --> 00:03:05,100 and we'll call the raise for status method. 38 00:03:05,370 --> 00:03:11,550 Just to make sure everything worked and nothing happened so that means there were no problems and no 39 00:03:11,550 --> 00:03:13,320 exceptions were raised by this function. 40 00:03:13,320 --> 00:03:20,070 So that means there were no problems downloading this website and now we'll have to call the B.S. for 41 00:03:20,070 --> 00:03:21,210 modules. 42 00:03:21,390 --> 00:03:25,260 Beautiful soup. 43 00:03:25,260 --> 00:03:26,950 Function and pass it. 44 00:03:27,030 --> 00:03:33,630 The H.T. email text that we've downloaded this will be in Razdan text and this will return a beautiful 45 00:03:33,630 --> 00:03:34,440 soop object. 46 00:03:34,440 --> 00:03:37,140 So let's just say that no variable named soup 47 00:03:40,330 --> 00:03:42,450 works. 48 00:03:43,650 --> 00:03:46,530 Be sure to spell beautiful soup correctly. 49 00:03:46,530 --> 00:03:52,140 No warning will show up because there have been some additions to the beautiful soup module and in fact 50 00:03:52,170 --> 00:03:56,020 it can parse HTL but also several other different things. 51 00:03:56,130 --> 00:04:01,790 So if we all assume that we do want to parse HMO when we call this beautiful suit function. 52 00:04:01,860 --> 00:04:05,490 Now this warning won't crash our program or anything it just looks really ugly. 53 00:04:05,490 --> 00:04:13,680 So to hide this we would just have to pass HVM Eldar parser as the second argument and this will tell 54 00:04:13,680 --> 00:04:14,280 beautiful suit. 55 00:04:14,280 --> 00:04:15,930 Yes we do want to pass. 56 00:04:15,960 --> 00:04:18,190 We do want to parse HMO. 57 00:04:18,360 --> 00:04:23,160 You could also not pass this and just simply ignore this ugly warning. 58 00:04:23,730 --> 00:04:32,130 So a warning is not an exception in Python so this soop object that we have is now ready to find HMO 59 00:04:32,130 --> 00:04:35,550 elements in the web page that we downloaded. 60 00:04:35,700 --> 00:04:42,540 The main way we do this is with the Select method and we pass it is string of the CSSA selector for 61 00:04:42,540 --> 00:04:45,280 the element or elements that we want to look at. 62 00:04:45,470 --> 00:04:46,880 A selector syntax. 63 00:04:46,890 --> 00:04:53,070 If you've ever done CSSA before you've already UCSC selectors they're kind of like regular expression 64 00:04:53,070 --> 00:05:00,840 syntax but CSSA selectors answer the question of how do I specify a particular part of the H.T. email 65 00:05:01,080 --> 00:05:03,010 document that I want to look at. 66 00:05:03,180 --> 00:05:08,360 And you could look at the HMO yourself and figure it out if you've already learned this syntax. 67 00:05:08,370 --> 00:05:10,950 Or you can have the browser figure it out for you. 68 00:05:10,950 --> 00:05:17,710 So let's just right click on this price information right here and select inspect element so this will 69 00:05:17,710 --> 00:05:26,020 show you that this spanne element is the HMO element that contains that price text that $22 and 86 cents 70 00:05:26,020 --> 00:05:26,440 . 71 00:05:26,440 --> 00:05:32,530 So to get a CSA a selector for this span element you can right click on the element and select copy 72 00:05:32,530 --> 00:05:33,640 CSSA path. 73 00:05:33,640 --> 00:05:40,390 If you use browsers other than Chrome there might be copies CSS selector or copy unique selector or 74 00:05:40,390 --> 00:05:45,550 something like that but just copy it and that'll be copied to the clipboard and then paste it into our 75 00:05:45,550 --> 00:05:46,200 code. 76 00:05:46,420 --> 00:05:53,440 And so this is the address of that span element that contains the price and what select does is it returns 77 00:05:53,470 --> 00:05:58,150 a list of all these element objects for all the matching elements. 78 00:05:58,150 --> 00:06:03,730 And since this is a unique selector that we gave there only be one element in this list. 79 00:06:03,730 --> 00:06:07,700 So let's save this to a variable called Elam's. 80 00:06:07,780 --> 00:06:11,920 So Elam's is just a list that contain one element object. 81 00:06:11,920 --> 00:06:16,010 So we'll just use the zero index to get that element. 82 00:06:16,240 --> 00:06:23,620 And each of these element objects just like the request says response objects as a member variable called 83 00:06:23,650 --> 00:06:31,180 text and text just holds a string value of the text inside that HTL ASTM l element. 84 00:06:31,180 --> 00:06:36,560 You can see right here has a bunch of new lines and the price inside of this. 85 00:06:36,640 --> 00:06:41,410 You can see this right here there's a bunch of tab characters and new lines and just whitespace characters 86 00:06:41,690 --> 00:06:42,140 . 87 00:06:42,220 --> 00:06:43,330 So that's kind of messy. 88 00:06:43,330 --> 00:06:47,260 Let's just let's call this strip string method. 89 00:06:47,410 --> 00:06:53,470 We've seen that before and this will return a new string without all of this whitespace characters on 90 00:06:53,530 --> 00:06:55,490 either side of the string. 91 00:06:55,660 --> 00:07:00,320 And that risk returns $22 86 cents. 92 00:07:00,370 --> 00:07:05,580 So we've just written Python code that downloads a Web site and finds information for us. 93 00:07:05,830 --> 00:07:08,840 Let's put all of this together inside of a single program. 94 00:07:08,890 --> 00:07:12,960 I'm going to click on File new file. 95 00:07:13,030 --> 00:07:18,350 Let's take this from the top so we're going to have to import this for and also request. 96 00:07:18,760 --> 00:07:22,990 Let's create a function called Get Amazon price. 97 00:07:22,990 --> 00:07:30,730 We just passed this a string of the products you sell and this function will return the price of the 98 00:07:30,730 --> 00:07:31,150 product. 99 00:07:31,150 --> 00:07:43,420 So when we call this later on say get Amazon price we pass it the or else the product we want and this 100 00:07:43,420 --> 00:07:47,020 will return some price information or store then a price variable. 101 00:07:47,410 --> 00:07:51,880 And then we can do whatever we want with that price information I'll just print it out to the screen 102 00:07:51,890 --> 00:07:52,160 . 103 00:07:52,350 --> 00:07:54,150 So the price is. 104 00:07:54,700 --> 00:08:02,230 And you stream concatenation just concatenate this and then print it out. 105 00:08:02,350 --> 00:08:05,780 So we want this function to return a string of the price. 106 00:08:05,800 --> 00:08:10,120 That's pretty simple That's basically all the steps that we've done already flutes put them all into 107 00:08:10,120 --> 00:08:10,800 this function. 108 00:08:10,810 --> 00:08:18,940 So first we use requests to download that page and we just call raise for status. 109 00:08:18,940 --> 00:08:24,130 So if there was some sort of problem downloading this it will raise an exception and crasser program 110 00:08:24,130 --> 00:08:24,870 . 111 00:08:24,910 --> 00:08:32,290 Next we want to create a super object by calling the beautiful function and passing it the H DML text 112 00:08:32,290 --> 00:08:34,070 of that web page we've downloaded that. 113 00:08:34,210 --> 00:08:38,920 Be in the text variable of the response object. 114 00:08:39,430 --> 00:08:42,940 And then we'll call the Select method. 115 00:08:45,490 --> 00:08:52,360 We'll just pass it this CSSA selector which we originally got just from the web page by right clicking 116 00:08:52,360 --> 00:08:55,540 on that price selecting Inspectah element. 117 00:08:55,850 --> 00:09:00,350 Then right clicking on the element and getting the CSSA path for it 118 00:09:03,840 --> 00:09:07,750 will return a list of matching elements for that CSSA selector. 119 00:09:07,750 --> 00:09:14,460 We only want the first one in there and that element object will also just like help response objects 120 00:09:14,460 --> 00:09:14,580 . 121 00:09:14,580 --> 00:09:19,780 I happened to also have a member variable called text that contains that content element objects also 122 00:09:19,780 --> 00:09:25,730 have a variable called Text coincidentally that contains the text inside that HTML. 123 00:09:25,900 --> 00:09:31,690 And it had a lot of whitespace on either side of that source just call the strip string method on the 124 00:09:31,690 --> 00:09:33,820 string inside of text. 125 00:09:33,820 --> 00:09:35,020 And so that'll give us that 126 00:09:37,510 --> 00:09:39,700 $22 a string. 127 00:09:39,730 --> 00:09:45,790 So we want our function to just return that floats go ahead and test this out. 128 00:09:45,790 --> 00:09:46,750 Oh wait. 129 00:09:46,750 --> 00:09:52,630 Almost forgot this beautiful soupfunction just get me just to make sure that that ugly warning message 130 00:09:52,630 --> 00:09:55,780 doesn't show up or to pass HTML parser to it. 131 00:09:55,920 --> 00:09:57,190 All right so let's test this out. 132 00:09:57,190 --> 00:10:04,850 I'm going to save this as an example that high and I'll press F5 to run it. 133 00:10:07,030 --> 00:10:08,090 And that worked. 134 00:10:08,530 --> 00:10:13,450 So behind the scenes of this really simple print function call it just displaces text. 135 00:10:13,450 --> 00:10:19,690 Our program went to the Amazon Web site downloaded the web page it parsed the HQ mail of that web page 136 00:10:19,690 --> 00:10:20,770 looking for the price. 137 00:10:21,040 --> 00:10:26,140 And then that function returned this price which we then just pass it along to print to print out the 138 00:10:26,140 --> 00:10:29,470 price of that is $22 86 cents. 139 00:10:29,530 --> 00:10:35,410 And just imagine if you had a hundred things that you needed to check the price on every day just to 140 00:10:35,410 --> 00:10:36,940 see if they changed or not. 141 00:10:37,060 --> 00:10:43,030 If you did that by hand you would have to go through every single product on this website over and over 142 00:10:43,030 --> 00:10:43,090 . 143 00:10:43,120 --> 00:10:49,020 Writing down all the prices that would be a bit of a pain but now we have a function we just pass it 144 00:10:49,020 --> 00:10:55,270 the or else we could just run this program to download all these hundreds or even thousands of product 145 00:10:55,540 --> 00:10:58,960 prices and just look at the prices there. 146 00:11:00,220 --> 00:11:06,250 So if we had hundreds or thousands of product you or els would just call get an Amazon price for each 147 00:11:06,250 --> 00:11:07,930 of them and that would return the price for us. 148 00:11:07,930 --> 00:11:12,030 We wouldn't have to touch the web browser ourselves at all. 149 00:11:12,070 --> 00:11:16,040 Now this code won't work on every Amazon page. 150 00:11:16,090 --> 00:11:23,470 It's very dependent on exactly how Amazon's format formats the team of their web pages if they ever 151 00:11:23,500 --> 00:11:28,420 update their web page or even just have the different kinds of products with different kinds of product 152 00:11:28,420 --> 00:11:29,160 web pages. 153 00:11:29,200 --> 00:11:35,900 Sometimes this code might fail we'd have to figure out the new CSSA selector for that new web page. 154 00:11:36,430 --> 00:11:42,400 So you can try adding a try and accept block to this code to gracefully handle any errors that come 155 00:11:42,400 --> 00:11:49,240 up or maybe even just put a debugger breakpoint to see exactly how this failed but even still this code 156 00:11:49,240 --> 00:11:50,830 will mostly work. 157 00:11:50,830 --> 00:11:53,530 So now go to some other Web sites and try scraping. 158 00:11:53,680 --> 00:12:03,120 I really recommend the National Weather Service hot weather without gov or the web comic Xscape CD. 159 00:12:03,160 --> 00:12:09,610 You could write some program that would figure out the email for this image right here and then use 160 00:12:09,610 --> 00:12:15,490 requests to download this image and then start parsing for this link information to figure out the URL 161 00:12:15,490 --> 00:12:18,440 of the previous Canek and then just repeat that. 162 00:12:18,460 --> 00:12:23,920 And that way you can repeatedly download all of the images off of this Web site. 163 00:12:24,850 --> 00:12:32,350 So to recap web pages or plain text files formatted as HVM male and female can be parsed with the beautiful 164 00:12:32,350 --> 00:12:39,350 suit much more beautiful soup is imported with the name B-S for you'll have import as far as your code 165 00:12:40,180 --> 00:12:46,120 so you can pass the string with the HTML to the bs4 for the beautiful soupfunction to get a super object 166 00:12:46,380 --> 00:12:52,990 and a super object has a Select method that can be passed a string of the CSSA selector for an HMO element 167 00:12:52,990 --> 00:12:53,730 . 168 00:12:53,760 --> 00:13:00,070 Now you can figure out the CSA select or string yourself if you know the selector string syntax but 169 00:13:00,100 --> 00:13:04,870 an easier way to do it is just use the browser's developer tools by right clicking in element and selecting 170 00:13:05,500 --> 00:13:11,860 copy CSSA path and the Select method will return a list of matching element objects. 171 00:13:11,860 --> 00:13:18,740 Each of these elements objects has a text variable with a string of the HQ of the elements HD and now 172 00:13:19,510 --> 00:13:23,330 each element object has a text variable with a string of that elements. 173 00:13:23,370 --> 00:13:24,100 HVM L