1 00:00:00,030 --> 00:00:05,520 Hey there! I'm glad if you're watching this! Is great that you have made it this 2 00:00:05,520 --> 00:00:10,769 far in the course and in this lecture actually throughout the next few 3 00:00:10,769 --> 00:00:17,699 lectures you'll learn how to scrap data from this website. So this is a real 4 00:00:17,699 --> 00:00:25,140 estate website and it what it does it's list properties for sale or for rent so 5 00:00:25,140 --> 00:00:32,189 basically let's say Rock Springs. There are actually a few of Rock Springs there so 6 00:00:32,189 --> 00:00:36,360 let's say Wyoming, Rock Springs in Wyoming. So we are looking for some 7 00:00:36,360 --> 00:00:39,230 properties there… 15 8 00:00:55,440 --> 00:01:01,410 …in Wyoming and it says it found 28 listings, so it's quite a small city, 9 00:00:58,920 --> 00:01:04,890 listings, so it's quite a small city, a small town. So the idea is that you'll learn 10 00:01:04,890 --> 00:01:13,140 how to scrap data of each of these properties so which can be prized. 11 00:01:13,140 --> 00:01:17,100 You have the address there and the number of beds that the property has, and 12 00:01:17,100 --> 00:01:24,600 the baths and so on. And you also get the square feet of the properties if that is 13 00:01:24,600 --> 00:01:30,149 available, some properties don't have that. So we have to count for that too. 14 00:01:30,149 --> 00:01:38,160 And also you will scrap data from multiple pages so we have we have 10 pages, 15 00:01:38,160 --> 00:01:42,060 10 properties here in this first page and then in the next page we have 10 others, 16 00:01:42,060 --> 00:01:47,520 and then in the third, the last page we have the rest which should be 8. 17 00:01:47,520 --> 00:01:53,310 So 28 in total. Now normally I assume that you know about the Requests and the 18 00:01:53,310 --> 00:01:57,270 Beautiful Soup libraries so you should have taken the previous lectures where 19 00:01:57,270 --> 00:02:03,630 we scrapped data for a simple webpage so that was a trivial example and I 20 00:02:03,630 --> 00:02:10,740 believe after that example you end up with that. And now what? So for 21 00:02:10,740 --> 00:02:17,760 that reason I want you to learn how to scrap some real data so this is one of 22 00:02:17,760 --> 00:02:22,110 the real world programs that we are building in this course, and in the script that we 23 00:02:22,110 --> 00:02:26,849 are about to write you face real programming issues there which is very 24 00:02:26,849 --> 00:02:33,750 important to build up your skills, your Python skills. Just one issue. Before 25 00:02:33,750 --> 00:02:38,970 scrapping data from our website it's good to read the data policies of that 26 00:02:38,970 --> 00:02:44,069 website so they may have some policies against the using or getting their data. 27 00:02:44,069 --> 00:02:48,720 I'm using this for educational purposes so I believe you'll be 28 00:02:48,720 --> 00:02:52,040 doing the same so that shouldn't be a problem. 29 00:02:52,040 --> 00:03:01,129 So let's go ahead and write that program. And I'll be using the Jupiter notebook. 30 00:03:01,129 --> 00:03:04,550 I suggest you do the same. So this will create a Jupiter notebook file, 31 00:03:09,769 --> 00:03:19,269 so I'm using Python 3. Call this century 21. Great, so you now know that the very 32 00:03:24,140 --> 00:03:30,709 first thing you want to do when writing a program is maybe import the libraries 33 00:03:30,709 --> 00:03:35,299 that you will be using, so we'll be using Request and give Beautiful Soup. 34 00:03:35,299 --> 00:03:45,500 So from bs4 import. Great, now can go to the next line and let's go back to 35 00:03:45,500 --> 00:03:52,609 the website. So now the first thing you may want to think about is how do we 36 00:03:52,609 --> 00:03:58,670 load the source code of the web pages. And now actually this is a bit complicated. 37 00:03:58,670 --> 00:04:04,310 I mean not much complicated, but it's different from the static web 38 00:04:04,310 --> 00:04:10,970 page that we scrapped in the previous lectures, so here we have we're going to 39 00:04:10,970 --> 00:04:16,039 be scrapping three pages as I said and the good thing is that every one of 40 00:04:16,039 --> 00:04:24,190 these pages has a unique URL so when you when you are in the main page here 41 00:04:24,190 --> 00:04:31,160 you see the URL is just plain simple but then when you search a place, so look at 42 00:04:31,160 --> 00:04:37,940 the URL now when I search the URL will change, so it went to real estate Rock 43 00:04:37,940 --> 00:04:44,840 Springs and Wyoming so this is the string so for the place that you 44 00:04:44,840 --> 00:04:51,080 searched, Rock Springs and WY, and you also got something else here that you 45 00:04:51,080 --> 00:04:58,880 need to be aware of, so the idea is that now you get this URL, so first 46 00:04:58,880 --> 00:05:05,000 we'll be scrapping the first page only and once we will grab that, then we will 47 00:05:05,000 --> 00:05:09,080 think about the next pages. So let's, because the next pages are the 48 00:05:09,080 --> 00:05:14,479 same, so the structure is the same, but we will just be writing a loop to iterate 49 00:05:14,479 --> 00:05:18,520 through the next pages. So let's go ahead and load the first 50 00:05:18,520 --> 00:05:26,759 page, say request dot get. Here's the URL 51 00:05:28,820 --> 00:05:37,970 and we want the content of this requests object so r.content and let's 52 00:05:37,970 --> 00:05:47,780 print it out, so a simple test there. Alright so my internet connection is 53 00:05:47,780 --> 00:05:54,920 working that's all we know about this code because we can't read it so what we 54 00:05:54,920 --> 00:06:01,100 want to do is you can go to the next cell and make that code more readable. 55 00:06:01,100 --> 00:06:05,950 So we need to use the Beautiful Soup library here. C and parser which is HTML parser and 56 00:06:14,510 --> 00:06:27,220 maybe print out soup.prettify. Let's see, so this is the page. 57 00:06:27,220 --> 00:06:33,430 Sometimes you may get kicked out of the web page so you may want to make sure 58 00:06:33,430 --> 00:06:41,870 that the page has loaded correctly so maybe could go here and search for 59 00:06:41,870 --> 00:06:50,780 something so Winchester. I'll search for Winchester. Yeah, so the page seems to 60 00:06:50,780 --> 00:06:57,590 have loaded correctly and we don't need the prettify so let me clean the 61 00:06:57,590 --> 00:07:03,230 notebook there. So we were able to load the page correctly with requests, and now 62 00:07:03,230 --> 00:07:07,490 what is next? Well next is we need to understand the structure of the web page 63 00:07:07,490 --> 00:07:13,700 so we need to use the inspect tool in our browser and we'll do that in the next 64 00:07:13,700 --> 00:07:15,000 lecture, so see you.