1 00:00:02,340 --> 00:00:09,820 Let's close this section on Pandas by looking at a real world example and this example 2 00:00:09,830 --> 00:00:16,690 what we're going to do is we're going to grab the address of each of these rows that we have in the 3 00:00:16,810 --> 00:00:17,940 data frame. 4 00:00:18,260 --> 00:00:25,580 And we're going to convert it to latitude and longitude coordinates so geographical coordinates in other 5 00:00:25,580 --> 00:00:31,210 words. Now addresses they define unique points on the Earth. 6 00:00:31,220 --> 00:00:39,250 So if you pass this address to some service let's say you throw it on Google Maps and Google Maps will 7 00:00:39,430 --> 00:00:44,090 point to you so it will generate a marker that will tell you where this point is located. 8 00:00:44,350 --> 00:00:50,650 Now on the background that marker actually has a pair of latitude and longitude coordinates and 9 00:00:50,650 --> 00:00:54,640 every point on Earth has these pair of coordinates. 10 00:00:54,640 --> 00:01:01,330 And if you follow the section on building web maps with Folium in the course and then you know what these 11 00:01:01,540 --> 00:01:04,490 latitudes and longitudes, how they come in handy 12 00:01:04,510 --> 00:01:10,400 when you build applications such as the web maps or other maps. 13 00:01:10,400 --> 00:01:15,940 Now the process to convert from addresses to coordinates is called geocoding. 14 00:01:15,940 --> 00:01:22,990 And if you want to convert from latitudinal and longitude to adresses that is called the reverse geocoding. 15 00:01:23,060 --> 00:01:26,640 In this lectur we'll going to be looking at geocoding. 16 00:01:26,650 --> 00:01:34,000 So basically what we're going to do is we'll add here a column to the data frame, actually two columns one 17 00:01:34,000 --> 00:01:37,960 for latitude and longitude for each of the rows. 18 00:01:38,380 --> 00:01:40,720 Now pandas can not do that directly. 19 00:01:40,930 --> 00:01:41,870 So you need the help 20 00:01:41,890 --> 00:01:52,780 of another library that is called Geopy and you can install Geopy with pip, so pip install and 21 00:01:52,780 --> 00:01:56,800 Geopy and just wait a while. 22 00:01:57,450 --> 00:01:59,710 Oh great. 23 00:01:59,860 --> 00:02:09,000 And then you know before going ahead and applying the geocoder to my data frame values I'd like to actually 24 00:02:09,270 --> 00:02:17,430 convert a single address, an address string with geocoder. Something you should be aware of is that to 25 00:02:17,430 --> 00:02:18,780 use Geopy, 26 00:02:18,900 --> 00:02:28,380 actually to use the Geocoder which is you know if you say import Geopy, and then if you say Geopy 27 00:02:29,320 --> 00:02:36,770 like that you'll see that you have a geocoder module among them which is this one here. 28 00:02:36,900 --> 00:02:42,250 Geocoders actually, and for the Geocoders to work 29 00:02:42,270 --> 00:02:49,170 You need an internet connection because what Geocoders will do, is it will get your address and then it will send 30 00:02:49,180 --> 00:02:57,570 that to an online service that has all of these addresses in a database and then for your address it 31 00:02:57,570 --> 00:03:00,990 will calculate the corresponding latitude and longitude values. 32 00:03:01,110 --> 00:03:03,560 So you need an internet connection. 33 00:03:04,100 --> 00:03:16,110 And yeah what you normally do is you know you want to import from geopy.geocoders import... 34 00:03:16,200 --> 00:03:22,960 Actually there are a few Geocoders there but we'll use Nominatim. 35 00:03:31,640 --> 00:03:39,030 And then what you do is you know you create a nominatim variable object. 36 00:03:39,030 --> 00:03:40,950 So you store that object in a variable. 37 00:03:46,980 --> 00:03:55,090 And once you have that object, then you pointed to the geocode method of the Nominatim object and 38 00:03:55,090 --> 00:03:57,700 you pass an address as a string. 39 00:03:57,710 --> 00:04:01,430 in there, let's say 3995 40 00:04:01,590 --> 00:04:11,260 23rd and then maybe the city and the zip code 94114. 41 00:04:11,630 --> 00:04:12,020 And if you 42 00:04:12,070 --> 00:04:17,880 execute that, you'll get a location data type there. 43 00:04:18,290 --> 00:04:23,800 So all these, and what that includes is you know it includes the address that you passed there. 44 00:04:24,180 --> 00:04:30,630 So this one and it has also added United States of America so the country in here. 45 00:04:31,830 --> 00:04:37,190 And you also get the latitude and longitude. And this one here, just ignore that. 46 00:04:37,230 --> 00:04:43,050 It's just a response from the geocoder so it doesn't mean much. 47 00:04:43,050 --> 00:04:45,160 Now sometimes though, it's rare 48 00:04:45,420 --> 00:04:48,600 but sometimes you may get a None object. 49 00:04:48,600 --> 00:04:56,940 So for instance if you pass the address which probably is not a real address I'm not sure about that. 50 00:04:56,940 --> 00:05:00,750 I don't know but if you say San Francisco 51 00:05:03,480 --> 00:05:10,860 CA 94119, if you expect that nothing will happen. 52 00:05:11,250 --> 00:05:21,350 And actually you can see that if you store this in a variable and then print n this will say that 53 00:05:21,380 --> 00:05:24,600 it's a None object so it doesn't have anything inside. 54 00:05:24,740 --> 00:05:32,590 So be aware of these scenarios as well. And yeah we had our working address, this one here. 55 00:05:33,200 --> 00:05:44,350 Let's store that in this variable and once you have that to extract the latitude and longitude you apply 56 00:05:44,360 --> 00:05:53,440 latitude for the latitude value and longitude for longitude, and that should do it because you know n, 57 00:05:53,910 --> 00:05:59,790 type(n), n is a special object, it's called a location object of geopy. 58 00:06:00,110 --> 00:06:08,150 So you need to apply those methods and that's how you convert an address string to a location or to latitude 59 00:06:08,150 --> 00:06:09,360 and longitude values. 60 00:06:09,470 --> 00:06:15,750 But how about converting an entire column of a data frame into latitude and longitude? 61 00:06:17,360 --> 00:06:31,400 So we've got this data frame. Df=pandas.read-csv. Super dot csv and this should be an underscore. 62 00:06:32,460 --> 00:06:39,010 Let me import Pandas first and print out the data frame. 63 00:06:39,050 --> 00:06:41,180 So this is our new data frame. 64 00:06:41,360 --> 00:06:43,770 Actually this is the old one that we've been using. 65 00:06:43,880 --> 00:06:52,770 We have five, six addresses there, six rows with and address, a city, and state, and country. 66 00:06:53,300 --> 00:07:01,270 Now the Geocode method more or less it accepts this kind of format so it expects from you 67 00:07:01,560 --> 00:07:09,830 the road name in here, and than the city, than zip code in here and the your country. 68 00:07:09,830 --> 00:07:17,010 So what we can do is we need to construct such a column in our data frame first. 69 00:07:17,140 --> 00:07:21,860 And yeah you can either create a new column or you can edit an existing one. 70 00:07:22,010 --> 00:07:29,930 So let's say I added the address, the existing address column so that would be equal to df address. 71 00:07:30,170 --> 00:07:37,130 So this value in here plus I'll need a coma in there. 72 00:07:37,850 --> 00:07:44,160 So a coma and maybe a space and plus df city. 73 00:07:44,300 --> 00:07:52,970 So a comma between address and city and then another comma and than plus df. 74 00:07:53,380 --> 00:08:08,020 State again and yet another comma like that, plus again df and lastly country. That should do it df. 75 00:08:10,600 --> 00:08:15,200 And yeah, we've got a complete address column in there. 76 00:08:15,430 --> 00:08:16,870 Great. 77 00:08:17,340 --> 00:08:24,300 And now we need to send this string to the geocode method. 78 00:08:25,200 --> 00:08:27,870 We need to do it for all the rows. 79 00:08:28,180 --> 00:08:31,860 You're probably thinking of iterating, but with pandas 80 00:08:31,880 --> 00:08:38,030 actually you don't need to iterate. Pandas is designed in a way that it allows you, 81 00:08:38,030 --> 00:08:47,600 it has some methods that allows you to apply a method or a function to all the rows of the data frame without 82 00:08:47,600 --> 00:08:57,260 having to write a for loop and to do that you know you need to create a new column, let's call 83 00:08:57,260 --> 00:09:01,020 it coordinates where you'd store the strings. 84 00:09:01,640 --> 00:09:05,150 You know this string in here. 85 00:09:05,300 --> 00:09:11,550 So this actually is not a string it's a location object but you can store it in your data frame. 86 00:09:11,960 --> 00:09:15,590 So we need to store locations for each of the rows. 87 00:09:15,650 --> 00:09:25,060 The way you do that is you know you point to the column that you want to pass to your geocoder and then 88 00:09:25,080 --> 00:09:28,390 you use a pandas method called applied. 89 00:09:28,880 --> 00:09:33,860 So what method do you want to apply to the values of the address column? 90 00:09:34,130 --> 00:09:42,960 Well that would be n, so n is the Nominatim object that we have here. 91 00:09:44,350 --> 00:09:57,590 Sorry, it's Nom, sorry! So that would be nom.geocode. And so the same as this one. 92 00:09:58,430 --> 00:10:00,040 So nom.geocode. 93 00:10:00,090 --> 00:10:05,150 But in this case you don't pass brackets there because the apply method 94 00:10:05,220 --> 00:10:06,670 will do it for you. 95 00:10:07,040 --> 00:10:13,700 So just like that and then you maybe you print out the data frame in there and see what you get. 96 00:10:18,590 --> 00:10:22,770 I got the service timed out geocoder is not working probably. 97 00:10:22,760 --> 00:10:25,290 Maybe I have a problem with my internet connection. 98 00:10:25,340 --> 00:10:32,960 So if you get this long error that is not your fault, it's a problem with geocoder with Geopy. 99 00:10:33,350 --> 00:10:35,020 I'll try that again. 100 00:10:38,090 --> 00:10:39,720 And yeah this time it worked. 101 00:10:40,080 --> 00:10:44,550 It was able to fetch the location objects in here. 102 00:10:44,990 --> 00:10:49,460 I mean you cannot see the latitude and longitude because it's a long string. 103 00:10:49,970 --> 00:11:00,960 But if you do it like that, coordinates, you you get the series for coordinates and yeah, that is not showing 104 00:11:00,960 --> 00:11:10,500 it either but you can do it's like you know df coordinates and then you access the first item only like 105 00:11:10,500 --> 00:11:15,780 that and then you get the entire text for the location. 106 00:11:15,780 --> 00:11:26,010 If you want the latitude, you get latitude only. And that brings us to the point that you may want to add another 107 00:11:26,070 --> 00:11:27,870 two columns in your dataframe 108 00:11:27,870 --> 00:11:32,180 where you fetch the latitude and longitude values. 109 00:11:32,220 --> 00:11:36,340 So our data frame is this one at the moment. 110 00:11:36,810 --> 00:11:47,070 And what you can do is you know you could create a latitude column in there. That would be good to df coordinates 111 00:11:48,650 --> 00:11:50,520 apply. 112 00:11:51,840 --> 00:11:55,280 You know you cannot apply latitude directly in there 113 00:11:57,960 --> 00:12:03,150 because you'll get this kind of error that says series has no attribute latitude. 114 00:12:03,270 --> 00:12:10,870 So you're applying a latitude method to a series but a series doesn't recognize that. What series recognizes 115 00:12:10,920 --> 00:12:13,270 is the apply method. 116 00:12:13,890 --> 00:12:21,820 So there you can't write your other methods. Now latitude will point to the values 117 00:12:21,820 --> 00:12:25,100 of these coordinates column. 118 00:12:25,320 --> 00:12:29,010 And in such scenario you use you use a lambda function 119 00:12:31,810 --> 00:12:35,170 which is an inline method to build the function. 120 00:12:35,250 --> 00:12:39,400 So you would say that lambda x, x is a temporary variable there. 121 00:12:39,670 --> 00:12:44,090 And you say x latitude. 122 00:12:44,410 --> 00:12:46,860 Let's keep it like that for now. 123 00:12:47,270 --> 00:12:51,910 Df thereand let's see what we got. 124 00:12:52,120 --> 00:12:56,580 Well ite say that's a non type object has no attributes latitude. 125 00:12:56,890 --> 00:13:01,460 So this can be quite tricky if you're not experienced with geocoding. 126 00:13:01,760 --> 00:13:11,430 And the reason we get this is that we have a None row, value in there among our rows andour row which 127 00:13:11,430 --> 00:13:20,410 is not a location data type does not have a latitude method because you know what we did here is we are 128 00:13:20,410 --> 00:13:22,460 storing all these values. 129 00:13:22,570 --> 00:13:24,310 So it is like a loop. 130 00:13:24,310 --> 00:13:30,400 We are storing all these values in the temporary x variable and than for each of these values we apply 131 00:13:30,400 --> 00:13:31,240 the latitude. 132 00:13:31,540 --> 00:13:39,190 So what Python will do is it will go through the first row and it will apply the latitude method to 133 00:13:39,190 --> 00:13:45,400 the first row and it will store it in the latitude column and then it goes to the second value but 134 00:13:45,430 --> 00:13:48,260 in this value latitude is not existing for None. 135 00:13:48,400 --> 00:13:51,030 So you get an error. To do that 136 00:13:51,070 --> 00:14:04,710 you could apply a conditional, an in line if conditional. You say if x is not None else None.Yeah, I know 137 00:14:04,820 --> 00:14:13,840 it's a bit confusing, but what we did is you know apply latitude if x is not None so it will apply 138 00:14:13,840 --> 00:14:17,700 this method for those rows, for those values. 139 00:14:18,690 --> 00:14:24,970 Otherwise it will store None in the current cell of the latitudes column. 140 00:14:25,700 --> 00:14:27,390 So I hope that is clear. 141 00:14:27,470 --> 00:14:29,370 I'll execute here. 142 00:14:30,260 --> 00:14:33,390 Yeah we got the latitude column in there. 143 00:14:33,950 --> 00:14:35,030 Great. 144 00:14:35,250 --> 00:14:37,530 And we can do the same for longitude. 145 00:14:37,530 --> 00:14:38,280 . 146 00:14:40,910 --> 00:14:42,800 Longitude. 147 00:14:43,030 --> 00:14:43,590 And here as well. 148 00:14:48,330 --> 00:14:49,460 Yeah, that was quick. 149 00:14:49,820 --> 00:14:50,690 And yeah, that's it. 150 00:14:50,780 --> 00:14:51,940 You have a latitude 151 00:14:51,940 --> 00:14:53,930 and a longitude column in your data frame. 152 00:14:54,310 --> 00:14:58,070 So please have a second look at what I wrote in here. 153 00:14:59,140 --> 00:15:04,190 And if there is something that doesn't make sense just drop a question and I'll be happy to answer you. 154 00:15:04,510 --> 00:15:09,270 So yeah we have quite a lot of flexibility working with data frames. 155 00:15:09,310 --> 00:15:10,880 I hope you enjoyed this. 156 00:15:11,040 --> 00:15:13,140 I'll talk to you in the next lectures. 157 00:15:13,380 --> 00:15:13,720 See you.