1 00:00:00,230 --> 00:00:07,140 So we've got the entire soup from the webpage which is actually the source 2 00:00:07,140 --> 00:00:13,799 code and we also got the source code of the divisions with class property row 3 00:00:13,799 --> 00:00:21,480 and now we were to grab the property price for only the first division so for 4 00:00:21,480 --> 00:00:32,250 only the first box here. Now to get all the boxes we need to iterate. So dd to 5 00:00:32,250 --> 00:00:43,050 delete that and for let's say item in all, okay, and actually I would like to keep 6 00:00:43,050 --> 00:00:48,899 the all printed out here so that you can see what all is, so if you forget all of 7 00:00:48,899 --> 00:00:53,399 that for a moment you're able to see what it is, so it's 8 00:00:53,399 --> 00:01:00,480 like a list, so for item in all let me go here so we'll be using the print 9 00:01:00,480 --> 00:01:05,909 method here, but I'd like to write it at the end because I don't want to add 10 00:01:05,909 --> 00:01:10,830 brackets now there, so it's good to have less brackets for now for you to 11 00:01:10,830 --> 00:01:17,700 see this more clear, the code, the actual code, so we are going through the first 12 00:01:17,700 --> 00:01:24,630 item which is this division here and we want to apply find all method to that 13 00:01:24,630 --> 00:01:32,340 item, or just the find method because we have only one h4 for here so find and we 14 00:01:32,340 --> 00:01:45,119 want to find h4 which has a class of propPrice, and then what we want to get 15 00:01:45,119 --> 00:01:50,790 from this product is the text and you saw that the text was a bit messy there 16 00:01:50,790 --> 00:01:58,140 so we want to replace the backslash Ns with nothing and then another 17 00:01:58,140 --> 00:02:04,829 replacement there, so there was space. Replace that with nothing so it's the same 18 00:02:04,829 --> 00:02:13,940 line as here. Then you want to print this out so add the print statement there. 19 00:02:13,940 --> 00:02:26,670 And yeah, that's it. Let's see how this goes. That was fast – 725, 452, yeah. That's 20 00:02:26,670 --> 00:02:36,900 good. All right, let me delete this to save some space, so we have all 21 00:02:36,900 --> 00:02:40,530 already defined here so we don't need to add it here. 22 00:02:40,530 --> 00:02:48,840 And let's go now and extract this zero gateway doesn't look a very good 23 00:02:48,840 --> 00:02:54,600 example, so let's ignore that and extract this address here, Winchester 24 00:02:54,600 --> 00:03:08,310 Boulevard and that would be Inspect, so that has a span tag with a class of proper dress 25 00:03:08,310 --> 00:03:15,600 collapse, but you can see that you also have another class here prop address 26 00:03:15,600 --> 00:03:20,880 collapse, so two classes. This is the IP address and this is the name of the town. 27 00:03:20,880 --> 00:03:29,900 And look if you have other prop address collapses there? No. No, we don't, so 28 00:03:29,900 --> 00:03:38,280 prop address collapse. Let's add another statement here. Item, so those dresses 29 00:03:38,280 --> 00:03:44,430 are inside this all method again so item find, now if you use find this time 30 00:03:44,430 --> 00:03:50,550 instead of find all, what the script will do is it will grab the first occurrence 31 00:03:50,550 --> 00:03:58,140 of prop address collapse, so it will give you the Winchester Boulevard address and it 32 00:03:58,140 --> 00:04:06,300 will ignore the second one, so it will ignore Rock Springs, so that means we need 33 00:04:06,300 --> 00:04:15,150 to use find all and if you want to get this first line, you'd want to pass 34 00:04:15,150 --> 00:04:21,860 the zero index to this result set list and then an index of one for the next line. 35 00:04:21,860 --> 00:04:33,640 So we'll do that. That was a span tag with a class of 36 00:04:33,640 --> 00:04:45,920 again prop address collapse, so let me print this out to see what we have 37 00:04:45,920 --> 00:04:55,220 this far. So here is a price of the first property and then we grab the results at 38 00:04:55,220 --> 00:05:02,930 list in here which actually has two elements so the address here and then 39 00:05:02,930 --> 00:05:09,890 the name of the town with a state code and zip code, so that means we need to 40 00:05:09,890 --> 00:05:21,920 apply here zero index if you want to get the first, okay, and then let's see. 41 00:05:21,920 --> 00:05:28,520 So here is the first one, the first element. If you want the second one you want to 42 00:05:28,520 --> 00:05:37,790 copy that and paste it here and pass one here, execute, and here we go. I guess you 43 00:05:37,790 --> 00:05:47,120 want the text so you want to apply the text method there. Great! And just for the sake 44 00:05:47,120 --> 00:05:53,270 of visibility let's add another line at the end of the loop to pass an empty 45 00:05:53,270 --> 00:05:59,780 string so now you can see the blocks more clear and we've got 10 blocks there. 46 00:05:59,780 --> 00:06:06,800 Quite awesome, isn't it? So let's go now and extract the number 47 00:06:06,800 --> 00:06:18,670 of beds and the number of baths, so that is a span tag with a class for bed, 48 00:06:22,230 --> 00:06:35,980 and baths. Here is the other span class. Info value for bath. Now here we got an 49 00:06:35,980 --> 00:06:42,520 issue. Now we need to be careful here because we need to grab the number of 50 00:06:42,520 --> 00:06:49,540 the beds here which is inside b tags. But the number of bath is also inside 51 00:06:49,540 --> 00:06:59,230 b tags, so you can go ahead and apply a find all method directly in this level, 52 00:06:59,230 --> 00:07:06,460 so directly under all, but that should probably cause some problems because you 53 00:07:06,460 --> 00:07:14,170 have multiple Bs there, so you've got also the area which should be 54 00:07:14,170 --> 00:07:23,110 a b tag as well, span class. Yeah, it's a b tag. So sometimes you have 3 b tags, sometimes 55 00:07:23,110 --> 00:07:30,340 you have more, so there may be two solutions there. You can either assume that beds are 56 00:07:30,340 --> 00:07:35,860 always the first ones and you can pass the index of zero to the results at list, 57 00:07:35,860 --> 00:07:41,980 so you get bets and then you get beds, and then full baths with an index of 1 and 2 58 00:07:41,980 --> 00:07:49,300 for these square feet here, and 3 for this, but I would prefer to do it in a more 59 00:07:49,300 --> 00:07:58,030 constructive way so I would like to go and extract these span tags first, and then 60 00:07:58,030 --> 00:08:04,420 I go inside each of the span tags and extract the value inside the B tag. 61 00:08:04,420 --> 00:08:15,660 So we've got span for info bed, and for full baths which is this one here. 62 00:08:15,660 --> 00:08:23,260 Actually this is listed as the third one, so beds are the first one, square feet, 63 00:08:23,260 --> 00:08:28,690 the area is the second, and full baths and so on. So we don't have to think about the 64 00:08:28,690 --> 00:08:35,430 order of them. We only need to grab the class names, so info bed. 65 00:08:35,589 --> 00:08:43,579 That means you want to access item find. Just find in this case, so you want 66 00:08:43,579 --> 00:08:56,829 to find the span and let's do first the beds which was info bed. 67 00:09:00,939 --> 00:09:13,370 So let's print that out first to see if we are good to go. So you get none for 68 00:09:13,370 --> 00:09:21,920 the first property, so this doesn't have any data for beds and baths, so you can't do 69 00:09:21,920 --> 00:09:27,800 anything about that. You just need to accept no data sometimes. And then here is 70 00:09:27,800 --> 00:09:33,110 the code, so the source code for the next one. Now here's a trick. If you apply the 71 00:09:33,110 --> 00:09:44,959 text method to this object so it should be here dot text, you'll get an error and look 72 00:09:44,959 --> 00:09:51,199 closely what it says, so it says none type object has no attribute text and that 73 00:09:51,199 --> 00:09:59,420 none type object was this here, so None. Which is a special object and it doesn't 74 00:09:59,420 --> 00:10:07,250 have a text method available so you need to count for that and the way you count 75 00:10:07,250 --> 00:10:16,420 for these exceptions is by using a try and accept statement, so you need to 76 00:10:16,420 --> 00:10:24,529 enclose this print statement here with try, so try this if it works, if it 77 00:10:24,529 --> 00:10:33,649 doesn't, do this other stuff here, so we just pass, so just ignore this none types there. 78 00:10:33,649 --> 00:10:45,500 So again try to get actually the text method out of these span tags, 79 00:10:45,500 --> 00:10:50,060 and if that it's not available then just pass to the next iteration of 80 00:10:50,060 --> 00:10:59,120 the loop, so let's try that. Yeah, now it's working. Now as you can see we got 81 00:10:59,120 --> 00:11:05,030 the text for the b tag, but also the types of the span tag so this is the 82 00:11:05,030 --> 00:11:12,200 text of the b tag, and this is the text of the span tag, so if that's fine for 83 00:11:12,200 --> 00:11:17,440 you, you can leave it like that, but I'd like to have the plain number only and 84 00:11:17,440 --> 00:11:30,260 to do that you need maybe apply the find method, so we have again, we have this 85 00:11:30,260 --> 00:11:35,630 source code so we need to find inside that source code, we need to find the 86 00:11:35,630 --> 00:11:42,110 b tag and which doesn't have a class name so we leave it like that and we 87 00:11:42,110 --> 00:11:48,550 need to extract the text out of that b tag and that gives you the number only. 88 00:11:48,550 --> 00:11:59,960 So 445 and so on, so that's good. Now let's do the same, so I'll copy this block 89 00:11:59,960 --> 00:12:13,430 and we'll do the same for the number of baths. Let's make a space here, and for the 90 00:12:13,430 --> 00:12:24,800 number of, that was so the area size and this another thing here, so half baths. 91 00:12:24,800 --> 00:12:31,700 We will count for that too. So info square feet, info square feet there. Would go 92 00:12:31,700 --> 00:12:44,530 here, like that and then we have info value full bath. 93 00:12:49,130 --> 00:13:08,400 And one more for the half baths, so that would be info value half bath. Execute. 94 00:13:08,400 --> 00:13:14,790 And we've got an indentation problem there. So try this one is good this one is not 95 00:13:14,790 --> 00:13:22,470 good, here as well, so we need to indent the statements after the try add the statement 96 00:13:22,470 --> 00:13:37,250 after accept, so execute and let's see. Good, so 4 beds and then 4 baths there, 97 00:13:37,250 --> 00:13:44,310 but then here we got the area of the property so the order is not very good 98 00:13:44,310 --> 00:13:53,430 like this. In that case we don't want to write pass here, but maybe we want 99 00:13:53,430 --> 00:14:05,660 to print none in those cases, so print none, print none, and same here. 100 00:14:05,660 --> 00:14:13,950 Execute and now it looks better, so now we know the order that, we know that we 101 00:14:13,950 --> 00:14:19,560 have 4 beds in this property, and the area size is not available, 102 00:14:19,560 --> 00:14:24,360 and then we have 4 baths, and then we don't have any information 103 00:14:24,360 --> 00:14:30,740 about the half baths there, so that is looking good. 104 00:14:30,770 --> 00:14:38,070 And I'd like to stop here and get some more data in the next lecture. 105 00:14:38,070 --> 00:14:39,000 Yeah, let's move on.