1 00:00:00,000 --> 00:00:08,940 Here we are again. We've got this script which produces this output so we have 2 00:00:08,940 --> 00:00:15,299 some attributes for each of these properties, and specifically those are the 3 00:00:15,299 --> 00:00:21,359 attributes of this block here, so price address and some attribites like the 4 00:00:21,359 --> 00:00:29,910 number of beds and so on. Now I would like to go further and extract the loot 5 00:00:29,910 --> 00:00:35,760 size for each property whenever that attribute is available so as you see 6 00:00:35,760 --> 00:00:41,879 here some properties don't have a lot size, and while lot size is an important 7 00:00:41,879 --> 00:00:47,789 attributes to get to know from a property, in this case is also quite 8 00:00:47,789 --> 00:00:56,730 a tricky one to extract from this webpage, and the reason is that if you look at 9 00:00:56,730 --> 00:01:03,510 the source code and you'll see that this is the name of the attribute so lot size, 10 00:01:03,510 --> 00:01:08,909 and this is the actual value that we want to extract, and these, both of these 11 00:01:08,909 --> 00:01:15,570 are inside this division, so the first line there. That division has a class name 12 00:01:15,570 --> 00:01:20,759 of column group. Now we have another column group class so another 13 00:01:20,759 --> 00:01:27,270 division with the column group class here is the second line, and then if you look 14 00:01:27,270 --> 00:01:35,780 at another property, so another row of property, you'll see that this column group 15 00:01:35,780 --> 00:01:47,369 division is repeating here, so if you write a loop to extract the feature, 16 00:01:47,369 --> 00:01:54,000 the span with feature name class for all the properties, for this property 17 00:01:54,000 --> 00:02:00,180 you'd be able to gather the actual lot size, but for this other one, you'd 18 00:02:00,180 --> 00:02:05,719 probably get the age of the property but you would be expecting the lot size. 19 00:02:05,719 --> 00:02:10,649 You'd be getting the age of the property because the age of the property 20 00:02:10,649 --> 00:02:16,100 is the first feature name there and lot size also is 21 00:02:16,100 --> 00:02:23,120 the first feature name so with feature name class in the second property, so 22 00:02:23,120 --> 00:02:26,840 this is one of those cases where you need to think about alternative 23 00:02:26,840 --> 00:02:33,580 solutions so if you point to column group method, so you say find all in 24 00:02:33,580 --> 00:02:39,050 division with class column group and inside that find the feature name with 25 00:02:39,050 --> 00:02:46,940 index 0 you would get the first lines. You know that. Now let's find the a solution then. 26 00:02:46,940 --> 00:02:57,650 Well how about looping through all these column groups, and then we check that if 27 00:02:57,650 --> 00:03:06,350 the text of the feature group is equal to lot size, then give me the text 28 00:03:06,350 --> 00:03:13,090 inside feature name. So in that specific iteration again if the feature group is 29 00:03:13,090 --> 00:03:20,660 equal to lot size, so it has the lot size string inside the text, then in that 30 00:03:20,660 --> 00:03:28,459 current iteration give me the text value of feature name, so we'll need a loop 31 00:03:28,459 --> 00:03:37,459 here inside our big loop, so this big loop here let's call it the big loop is 32 00:03:37,459 --> 00:03:46,910 going through each of these rows. And then inside that, inside the current 33 00:03:46,910 --> 00:03:52,220 iteration of that big loop, so let's say the current iteration would be going 34 00:03:52,220 --> 00:03:59,030 through this property so inside that we go to column group divisions, so let's say 35 00:03:59,030 --> 00:04:08,150 for, let's call this variable column group, in the current variable b item, 36 00:04:08,150 --> 00:04:11,980 so again item as we did there. Item dot find and find all actually so 37 00:04:18,620 --> 00:04:25,570 you want to find divisions with a class name of 38 00:04:25,570 --> 00:04:40,970 column group, and let me quickly print out the column group variable so just to 39 00:04:40,970 --> 00:04:48,530 see what we have so far. And here we go. So these are the data we have, and then 40 00:04:48,530 --> 00:04:58,280 here goes, up here, here are the source code for each of column group divisions. 41 00:04:58,280 --> 00:05:03,710 So the first division, the second division, and so on. So this is 42 00:05:03,710 --> 00:05:10,250 features which corresponds to this header here, and then we have a key 43 00:05:10,250 --> 00:05:16,330 architecture style which is not what we need. We have roof type. So at each iteration 44 00:05:16,330 --> 00:05:22,490 Python is printing out these lines. Now what we want to do with these lines, 45 00:05:22,490 --> 00:05:27,070 let's keep the print there for a while, what we want to do with these lines is 46 00:05:27,070 --> 00:05:31,550 iterate through them again, so we will iterate through this div and we will 47 00:05:31,550 --> 00:05:37,039 iterate through this other div and we will see that if the text of the 48 00:05:37,039 --> 00:05:43,039 feature group in that current iteration has the string lot size, then we will get 49 00:05:43,039 --> 00:05:49,789 the text of feature name for that current iteration so again, we need a for 50 00:05:49,789 --> 00:06:00,650 loop here which will go through each of these column group divisions so what it 51 00:06:00,650 --> 00:06:05,539 will do, is access the feature group element and the feature name element as 52 00:06:05,539 --> 00:06:14,240 well, so for feature group, feature name in, and we need to use the zip function 53 00:06:14,240 --> 00:06:19,639 there, so as you may remember it, zip function which is a built in Python 54 00:06:19,639 --> 00:06:25,789 function is used to iterate through two lists at the same time. So here is where 55 00:06:25,789 --> 00:06:30,500 you enter the two lists inside the zip function, and our two this would be, 56 00:06:30,500 --> 00:06:42,020 the first one would be column group dot find all spans, so span with 57 00:06:42,020 --> 00:06:51,999 a class name of feature group and similarly we need to access the column 58 00:06:51,999 --> 00:07:01,939 group dot find all and we need the span tag again, but this time we need a 59 00:07:01,939 --> 00:07:12,039 class of feature name, so don't confuse these variables with these class names. 60 00:07:12,039 --> 00:07:23,589 And here causes the zip function, so this bracket here, so after the for statement 61 00:07:23,589 --> 00:07:30,919 you need the colon here. Okay, now what do you want to do for each feature group? 62 00:07:30,919 --> 00:07:43,729 Let's temporarily print out the feature group text for instance and also the 63 00:07:43,729 --> 00:07:52,329 feature name dot text and let me delete this so this column group here 64 00:07:52,329 --> 00:07:59,509 corresponds with this block of divisions, so execute that. We've 65 00:07:59,509 --> 00:08:07,909 gotten invalid syntax here and look for the arrow. Yeah, here's a small arrow so 66 00:08:07,909 --> 00:08:14,599 this one here is pointing us to this character which actually is a semicolon. 67 00:08:14,599 --> 00:08:20,599 For some reason I put a semicolon so it should be a colon, not semicolon, okay. 68 00:08:20,599 --> 00:08:28,429 Let me execute that again. So here is what we get. For the first property for 69 00:08:28,429 --> 00:08:32,659 instance we got this previous attribute printed out, so none, none, none for this 70 00:08:32,659 --> 00:08:38,810 first one. And then we have, here is the feature group text and here is the 71 00:08:38,810 --> 00:08:43,669 feature name text, and similarly we have the feature group text here for the next 72 00:08:43,669 --> 00:08:49,740 feature group and feature name and so on. Then we have the next property there 73 00:08:49,740 --> 00:08:57,930 which actually has quite a lot of attributes there, heating fuel, gas and 74 00:08:57,930 --> 00:09:03,440 here is something you should know now. You should be aware about, let me open again 75 00:09:03,440 --> 00:09:16,410 the Century21 website. Rock Springs, so as you can see the second property doesn't have 76 00:09:16,410 --> 00:09:26,340 this many attributes, so it has age, appliances, basement, but not bath features, 77 00:09:26,340 --> 00:09:33,300 and cooling etc. So these extra attributes are somehow hidden in 78 00:09:33,300 --> 00:09:39,240 the source code of this webpage and you can access them if you go, so if you 79 00:09:39,240 --> 00:09:45,660 click the link of the web page, of the specific property, so here are all the 80 00:09:45,660 --> 00:09:51,600 attributes that we're seeing here. So fireplace count, 2 fireplaces. Here is 81 00:09:51,600 --> 00:09:58,830 fireplace count, and you can also see that the lot size is listed in the property 82 00:09:58,830 --> 00:10:05,940 webpage so here it is which is actually a good thing, so we are being able to 83 00:10:05,940 --> 00:10:12,360 extract all the attributes of a property without having to go inside the 84 00:10:12,360 --> 00:10:19,890 property webpage, the property link so that's a good thing. And now what we need 85 00:10:19,890 --> 00:10:28,020 to do is, let's keep the print statement there for a while and we need to check, so here 86 00:10:28,020 --> 00:10:40,740 under the for loop if lot size in feature group dot text, so for instance 87 00:10:40,740 --> 00:10:48,810 is the lot size string inside this feature group text, if it is then give me 88 00:10:48,810 --> 00:10:56,990 the feature name dot text, so it will give me the value here, if it is not then 89 00:10:56,990 --> 00:11:00,870 just don't do anything. So we leave the loop as it is. 90 00:11:00,870 --> 00:11:06,360 So let me give you this function now, the print function that we executed 91 00:11:06,360 --> 00:11:15,480 earlier and execute this. So here we go. We've got nothing here. 92 00:11:15,480 --> 00:11:21,480 These are previous attributes, and then we have four attributes here, 93 00:11:21,480 --> 00:11:27,470 and then we have 0.21 acres. Here we got half an acre. 94 00:11:27,470 --> 00:11:34,820 Okay, that is looking good now and I'd like now to go through the next lecture 95 00:11:34,820 --> 00:11:39,480 where you'll learn how to get rid of these print statements and to actually 96 00:11:39,480 --> 00:11:46,860 store all these attributes in a table, so in a Pandas data frame, then we export 97 00:11:46,860 --> 00:11:52,500 the data frame simply in a CSV file or excel file, wherever you like. So we're 98 00:11:52,500 --> 00:11:57,870 going to have like a price column there, an address column, a state column with 99 00:11:57,870 --> 00:12:05,760 a zip code, and also columns with these attributes and the lot size as well. 100 00:12:05,760 --> 00:12:06,000 So let's move on.