WEBVTT 1 00:00:00.400 --> 00:00:01.970 line:15% Hi guys, and welcome back. 2 00:00:01.970 --> 00:00:03.620 line:15% In this video, we're going to be looking 3 00:00:03.620 --> 00:00:05.440 at using the Chrome driver 4 00:00:05.440 --> 00:00:06.630 with our project, 5 00:00:06.630 --> 00:00:08.720 instead of the requests module. 6 00:00:08.720 --> 00:00:10.430 We're not going to change what the project does, 7 00:00:10.430 --> 00:00:12.360 we're not going to look at the new 8 00:00:12.360 --> 00:00:14.250 Javascript page yet. 9 00:00:14.250 --> 00:00:16.560 We're just going to use Chrome in here, 10 00:00:16.560 --> 00:00:18.400 so that we get accustomed to using it, 11 00:00:18.400 --> 00:00:19.350 and then we will move over 12 00:00:19.350 --> 00:00:21.090 to the new page. 13 00:00:21.090 --> 00:00:22.460 The first thing we have to do, 14 00:00:22.460 --> 00:00:25.020 is to get the Chrome driver. 15 00:00:25.020 --> 00:00:25.853 So we're going to do 16 00:00:25.853 --> 00:00:28.900 from.selenium import webdriver. 17 00:00:28.900 --> 00:00:30.750 We're no longer going to need requests, 18 00:00:30.750 --> 00:00:34.100 because the webdriver is going to be doing the requesting. 19 00:00:34.100 --> 00:00:36.180 So we no longer need that. 20 00:00:36.180 --> 00:00:37.610 Next, we're going to actually 21 00:00:37.610 --> 00:00:38.750 launch Chrome. 22 00:00:38.750 --> 00:00:39.583 So we're gonna do, 23 00:00:39.583 --> 00:00:40.416 say something like, 24 00:00:40.416 --> 00:00:42.567 chrome equals webdriver.Chrome, 25 00:00:43.940 --> 00:00:45.180 and here we're going to pass in 26 00:00:45.180 --> 00:00:48.420 the executable path, 27 00:00:48.420 --> 00:00:50.140 and that's going to be the path, 28 00:00:50.140 --> 00:00:53.130 to where your Chrome driver programme is. 29 00:00:53.130 --> 00:00:54.350 My case it's in, 30 00:00:54.350 --> 00:00:57.820 usr/local/bin/chromedriver, 31 00:00:57.820 --> 00:00:58.870 but if you put it somewhere else, 32 00:00:58.870 --> 00:01:00.860 then make sure that you put the correct path there, 33 00:01:00.860 --> 00:01:01.740 and if you're on Windows, 34 00:01:01.740 --> 00:01:03.630 make sure that you put the .exe there, 35 00:01:03.630 --> 00:01:05.590 if it's on the filename. 36 00:01:05.590 --> 00:01:08.085 That will actually launch Chrome, 37 00:01:08.085 --> 00:01:11.300 and it will give you this chrome variable, 38 00:01:11.300 --> 00:01:13.620 that you can use to interact with it. 39 00:01:13.620 --> 00:01:16.930 So now instead of making this request here, 40 00:01:16.930 --> 00:01:18.470 to get the HTML content, 41 00:01:18.470 --> 00:01:21.980 we're going to say, chrome.get, 42 00:01:21.980 --> 00:01:26.870 and then we're going to go to http://quotes.toscrape.com. 43 00:01:26.870 --> 00:01:29.570 Now instead of passing the page content, 44 00:01:29.570 --> 00:01:31.120 to our QuotesPage object, 45 00:01:31.120 --> 00:01:33.420 we're actually going to pass chrome, 46 00:01:33.420 --> 00:01:35.990 because this Chrome instance, 47 00:01:35.990 --> 00:01:37.940 is running the page. 48 00:01:37.940 --> 00:01:40.120 So it has access to everything inside the page, 49 00:01:40.120 --> 00:01:41.870 we no longer need to limit ourselves, 50 00:01:41.870 --> 00:01:44.660 to only getting the HTML content. 51 00:01:44.660 --> 00:01:47.130 We can actually pass the chrome object, 52 00:01:47.130 --> 00:01:48.030 to the QuotesPage, 53 00:01:48.030 --> 00:01:50.220 and use that chrome object inside 54 00:01:50.220 --> 00:01:52.140 this other object. 55 00:01:52.140 --> 00:01:54.250 For example, if we wanted to click around on things, 56 00:01:54.250 --> 00:01:55.270 or stuff like that, 57 00:01:55.270 --> 00:01:56.670 we can do that as long as we have 58 00:01:56.670 --> 00:01:58.010 this chrome object. 59 00:01:58.010 --> 00:02:00.280 If we pass in only the HTML content, 60 00:02:00.280 --> 00:02:01.950 we wouldn't be able to click around in it, 61 00:02:01.950 --> 00:02:04.400 because it will just be a string. 62 00:02:04.400 --> 00:02:06.350 Now notice that we have some red lines here, 63 00:02:06.350 --> 00:02:08.780 because we do have to instal selenium. 64 00:02:08.780 --> 00:02:11.400 So just go over to your Python preferences, 65 00:02:11.400 --> 00:02:15.010 and go to the project interpreter, 66 00:02:15.010 --> 00:02:18.750 click the plus icon, and instal selenium. 67 00:02:18.750 --> 00:02:19.620 Don't need anything else, 68 00:02:19.620 --> 00:02:21.550 just selenium is okay, 69 00:02:21.550 --> 00:02:22.890 and once it's installed, 70 00:02:22.890 --> 00:02:24.560 those red lines will go away. 71 00:02:24.560 --> 00:02:25.610 All right, that's your instal. 72 00:02:25.610 --> 00:02:27.850 So we can close and press okay, 73 00:02:27.850 --> 00:02:30.970 and the red lines go away so that's all good. 74 00:02:30.970 --> 00:02:33.100 Now we go over to the quotes page, 75 00:02:33.100 --> 00:02:37.000 because obviously this expects the HTML content 76 00:02:37.000 --> 00:02:38.110 to be passed to it, 77 00:02:38.110 --> 00:02:39.070 we need to change it, 78 00:02:39.070 --> 00:02:42.170 so that it expects now the chrome instance. 79 00:02:42.170 --> 00:02:43.050 So what we're going to do is, 80 00:02:43.050 --> 00:02:46.150 we're going to remove this dependency from BeautifulSoup, 81 00:02:46.150 --> 00:02:48.000 because we're no longer going to need that, 82 00:02:48.000 --> 00:02:50.840 we're going to use chrome and selenium themselves, 83 00:02:50.840 --> 00:02:52.570 to traverse the tree. 84 00:02:52.570 --> 00:02:54.763 So we don't need BeautifulSoup, 85 00:02:55.710 --> 00:02:58.863 and here we're going to say self.browser is browser, 86 00:03:00.760 --> 00:03:03.280 and we're going to pass that in as the parameter. 87 00:03:03.280 --> 00:03:04.710 Now this browser here, 88 00:03:04.710 --> 00:03:07.610 we can use to find things within the page. 89 00:03:07.610 --> 00:03:08.870 Selenium comes with that, 90 00:03:08.870 --> 00:03:10.780 so as soon you have your browser, 91 00:03:10.780 --> 00:03:12.260 you can do self.browser, 92 00:03:13.960 --> 00:03:17.627 .find_elements_by_css_selector, 93 00:03:18.560 --> 00:03:20.830 and then you pass in your selector there, 94 00:03:20.830 --> 00:03:22.233 which is this right here. 95 00:03:23.400 --> 00:03:26.160 Okay, now it's quite common when you have 96 00:03:26.160 --> 00:03:27.920 a long string like this one, 97 00:03:27.920 --> 00:03:31.440 to reformat it so that it is a little bit more readable, 98 00:03:31.440 --> 00:03:33.653 and we're going to do it like this. 99 00:03:34.850 --> 00:03:37.330 Okay, this is just a way to format this. 100 00:03:37.330 --> 00:03:39.770 If you are using a formatter like Black, 101 00:03:39.770 --> 00:03:40.920 which I recommend, 102 00:03:40.920 --> 00:03:42.370 then it will do this for you, 103 00:03:42.370 --> 00:03:44.990 but when you've got a longer list comprehension like that, 104 00:03:44.990 --> 00:03:46.705 it's normal to divide it into two, 105 00:03:46.705 --> 00:03:49.020 the thing that you want to put in the list, 106 00:03:49.020 --> 00:03:50.110 and then your loop, 107 00:03:50.110 --> 00:03:51.650 and then of course anything else, 108 00:03:51.650 --> 00:03:52.633 and also the index. 109 00:03:54.240 --> 00:03:56.160 Okay, now that we have our quotes, 110 00:03:56.160 --> 00:03:57.090 and again, this is just going to do 111 00:03:57.090 --> 00:03:58.010 the exact same thing, 112 00:03:58.010 --> 00:03:59.660 only instead of using BeautifulSoup, 113 00:03:59.660 --> 00:04:02.350 we're using the browser. 114 00:04:02.350 --> 00:04:05.220 We do have to go over to the quote parser, 115 00:04:05.220 --> 00:04:07.920 and make these changes as well. 116 00:04:07.920 --> 00:04:10.090 Instead of select one, 117 00:04:10.090 --> 00:04:10.950 what we're going to have to do, 118 00:04:10.950 --> 00:04:15.950 is again, find element by css selector. 119 00:04:16.460 --> 00:04:17.960 Notice that in the quotes page, 120 00:04:17.960 --> 00:04:20.980 we did find elements by css selector, 121 00:04:20.980 --> 00:04:22.380 here we only want one, 122 00:04:22.380 --> 00:04:26.060 so it is find element by css selector. 123 00:04:26.060 --> 00:04:27.140 Instead of .String, 124 00:04:27.140 --> 00:04:29.760 it is .text if you are using selenium. 125 00:04:29.760 --> 00:04:32.730 So yeah, we just copy that into here, 126 00:04:32.730 --> 00:04:35.710 and into here, but we don't want the text, 127 00:04:35.710 --> 00:04:37.730 and we also want multiple elements, 128 00:04:37.730 --> 00:04:38.810 so it's gonna be once again, 129 00:04:38.810 --> 00:04:41.090 find elements by css selector. 130 00:04:41.090 --> 00:04:44.100 The thing that we're interacting with now, 131 00:04:44.100 --> 00:04:45.770 when we use self.parent, 132 00:04:45.770 --> 00:04:48.320 or when we get these tags, 133 00:04:48.320 --> 00:04:51.250 is no longer a BeautifulSoup element, 134 00:04:51.250 --> 00:04:53.140 because we're not using BeautifulSoup. 135 00:04:53.140 --> 00:04:54.810 These things like self.parent, 136 00:04:54.810 --> 00:04:57.710 are all selenium objects, 137 00:04:57.710 --> 00:04:59.960 so they have different methods in them, 138 00:04:59.960 --> 00:05:02.020 and they have different properties in them, 139 00:05:02.020 --> 00:05:04.770 but overall their logic is the same. 140 00:05:04.770 --> 00:05:06.000 You're searching for them, 141 00:05:06.000 --> 00:05:07.330 and then you're getting the text out. 142 00:05:07.330 --> 00:05:09.453 It's just the names that have changed a little. 143 00:05:10.840 --> 00:05:12.350 If we go back to app.py, 144 00:05:12.350 --> 00:05:13.453 and we run it, 145 00:05:15.350 --> 00:05:18.080 you'll see that Chrome gets launched, 146 00:05:18.080 --> 00:05:20.190 and we load the page, 147 00:05:20.190 --> 00:05:22.720 and then we get our quotes down here, 148 00:05:22.720 --> 00:05:24.730 just as we did in the previous application, 149 00:05:24.730 --> 00:05:26.820 but now we do have Chrome here, 150 00:05:26.820 --> 00:05:27.653 and it tells you that, 151 00:05:27.653 --> 00:05:30.210 "Chrome is being controlled by automated test software." 152 00:05:30.210 --> 00:05:32.730 Remember what I said in the first video of this section, 153 00:05:32.730 --> 00:05:35.830 that the chrome driver is for testing websites, 154 00:05:35.830 --> 00:05:37.510 that's why it tells you that it's being controlled 155 00:05:37.510 --> 00:05:39.470 by automated test software, 156 00:05:39.470 --> 00:05:42.450 even though we're doing nothing to do with testing, 157 00:05:42.450 --> 00:05:44.933 but nonetheless, that's why selenium exists. 158 00:05:46.570 --> 00:05:50.610 All right, so that's how you use the Chrome web driver, 159 00:05:50.610 --> 00:05:52.750 you just ask it where you want to go, 160 00:05:52.750 --> 00:05:55.150 and then we're going to be passing it over to our page, 161 00:05:55.150 --> 00:05:56.410 and our parsers, 162 00:05:56.410 --> 00:05:57.640 and we're going to be using the methods 163 00:05:57.640 --> 00:06:00.070 that are defined inside selenium, 164 00:06:00.070 --> 00:06:02.763 to find elements and extract their contents. 165 00:06:03.760 --> 00:06:07.500 Now, because we have actual selenium objects, 166 00:06:07.500 --> 00:06:09.570 and the actual browser, 167 00:06:09.570 --> 00:06:11.270 we can click on things, 168 00:06:11.270 --> 00:06:13.050 for example on drop downs. 169 00:06:13.050 --> 00:06:14.700 So we're going to start looking, 170 00:06:14.700 --> 00:06:16.710 into how to do that in the next videos, 171 00:06:16.710 --> 00:06:18.950 as we move over to the new page 172 00:06:18.950 --> 00:06:20.680 that uses Javascript. 173 00:06:20.680 --> 00:06:22.230 I'll see you in the next video.