1

00:00:00,240  -->  00:00:07,680
Welcome to lesson 41 which roughly covers pages 256 261 of the automate the boring stuff with Python

2

00:00:07,710  -->  00:00:13,690
textbook in the previous two lessons you learned how to download web pages and parse each team out.

3

00:00:13,910  -->  00:00:19,070
This is fine if you just need the static text from you or else but sometimes the web pages you want

4

00:00:19,070  -->  00:00:25,560
to access rule require you to log in or they rely on Javascript in order to work properly.

5

00:00:25,560  -->  00:00:29,300
This isn't something you can easily do by just downloading a URL.

6

00:00:29,730  -->  00:00:35,040
The selenium module will launch a Web browser that you can programmatically control from your python

7

00:00:35,040  -->  00:00:36,180
program.

8

00:00:36,180  -->  00:00:42,480
You can call functions that will find each TNL in the browser or fill out forms and log in fields and

9

00:00:42,480  -->  00:00:43,870
click submit buttons.

10

00:00:44,030  -->  00:00:49,770
But because it launches a Web browser it's a bit slower and hard to run in the background if say you

11

00:00:49,770  -->  00:00:53,160
just need to download some files from the web.

12

00:00:53,160  -->  00:00:58,330
Selenium is a third party module that you can install by running Pipp install selenium.

13

00:00:58,410  -->  00:01:04,140
Consult the course notes or Appendix A of the automate book for instructions on installing it for this

14

00:01:04,140  -->  00:01:04,560
lesson.

15

00:01:04,560  -->  00:01:06,860
You'll need the Firefox web browser installed.

16

00:01:06,870  -->  00:01:08,650
This will be the browser that you control.

17

00:01:08,670  -->  00:01:11,810
If you don't already have Firefox you can download it for free from.

18

00:01:11,850  -->  00:01:19,340
Get Firefox dotcom importing the modules for selenium is slightly tricky instead of imports millennium

19

00:01:19,340  -->  00:01:19,530
.

20

00:01:19,590  -->  00:01:23,910
You'll need to run from selenium.

21

00:01:23,920  -->  00:01:30,210
Import web driver and the exact reason why this Lennier module is set up this way is kind of beyond

22

00:01:30,210  -->  00:01:31,700
the scope of this lesson.

23

00:01:31,710  -->  00:01:34,670
Just remember that you have to import it as from selenium.

24

00:01:34,710  -->  00:01:36,300
Import web driver.

25

00:01:36,300  -->  00:01:44,430
And after that you can launch the Firefox browser with selenium by calling the web driver dot Firefox

26

00:01:44,430  -->  00:01:50,040
function and this all will return a browser object which will store here in this variable called browser

27

00:01:50,050  -->  00:01:50,550
.

28

00:01:51,180  -->  00:01:58,710
But notice once I run this instruction a new Firefox window has been launched I'll just set this off

29

00:01:58,710  -->  00:02:04,190
to the side and you can now control this by calling methods on this browser object.

30

00:02:04,200  -->  00:02:08,830
Most basic of which is the get method which will just send this browser to a URL.

31

00:02:08,950  -->  00:02:17,730
So I'll just pass it the string automake the boring stuff dot com and you can see my Python code is

32

00:02:17,730  -->  00:02:25,820
controlling this browser so everything that a web browser does selenium is basically simulating.

33

00:02:25,830  -->  00:02:30,060
Now let's use CSSA selectors to find an element on this page.

34

00:02:30,090  -->  00:02:36,820
Say I wanted to click on this link right here for the Chapter 0 link.

35

00:02:36,870  -->  00:02:44,670
I'm just going to use the trick of right clicking on it and selecting inspect element and here all right

36

00:02:44,670  -->  00:02:51,600
click here to get a unique CSSA selector by clicking on copy unique selector.

37

00:02:51,600  -->  00:02:57,060
So this is a selector string just like the kind that we used in beautiful soup.

38

00:02:57,650  -->  00:03:09,080
So what I'm going to do to get this element is I will call browsers find element by CSX selector.

39

00:03:09,090  -->  00:03:15,420
Just kind of a long name for this method but I'll just pass it that CSSA selector.

40

00:03:15,770  -->  00:03:23,100
Now you can see I have a web element object stored in the elem variable and so once I have this element

41

00:03:23,100  -->  00:03:30,240
object which represents a single element on this web page I can then just call it click method and that

42

00:03:30,240  -->  00:03:34,440
will simulate clicking on that link on that element in the web browser.

43

00:03:34,440  -->  00:03:39,660
So if this element was a link or if it was a checkbox or if it was a submit button you can simulate

44

00:03:39,720  -->  00:03:41,610
clicking on it.

45

00:03:41,940  -->  00:03:46,620
Now you can also specify instead of a unique CSSA selector.

46

00:03:46,620  -->  00:03:54,930
A more general CSSA selector in that will match multiple elements and using that we can then call the

47

00:03:55,080  -->  00:04:05,010
find elements plural by CSSA selector and say if I just wanted to get all of the paragraph elements

48

00:04:05,160  -->  00:04:13,650
from that HTL page I can see that this method returns a list of these element objects.

49

00:04:13,650  -->  00:04:18,370
Here I have 109 matching paragraph elements from that page.

50

00:04:18,390  -->  00:04:23,980
Selenium has several different methods for getting web elements from the web page.

51

00:04:24,060  -->  00:04:30,120
The ones that you'll most often use are find element by CSSA selector and find elements by CSSA selector

52

00:04:30,120  -->  00:04:30,300
.

53

00:04:30,300  -->  00:04:36,710
But you can also find elements by class name by id link text the name or the tag name.

54

00:04:36,720  -->  00:04:42,240
All of these methods have a singular element form that will just return the first matching element or

55

00:04:42,240  -->  00:04:48,100
find elements plural which will return every matching element that it finds.

56

00:04:48,240  -->  00:04:50,010
And there is more documentation on these.

57

00:04:50,010  -->  00:04:53,110
In the automake book so we know how to click on things.

58

00:04:53,160  -->  00:04:56,600
Let's see how we can type into a web page as well.

59

00:04:56,640  -->  00:05:02,160
I'm just going to click on this link right here and here we can find a search field for the automate

60

00:05:02,160  -->  00:05:03,240
the boring stuff.

61

00:05:03,250  -->  00:05:04,390
Dotcom Web site.

62

00:05:04,500  -->  00:05:12,300
So I'm just going to right click on this to find its to find a CSSA selector that matches this just

63

00:05:12,480  -->  00:05:14,820
copy unique selector.

64

00:05:15,060  -->  00:05:24,120
So here I'll just have to get a new element for that search field by calling the find element by CSSA

65

00:05:24,120  -->  00:05:27,520
selector.

66

00:05:28,880  -->  00:05:31,260
And now I have this element object.

67

00:05:31,310  -->  00:05:33,940
Call It's send keys method.

68

00:05:34,280  -->  00:05:39,330
And then I can pass it any string which will then be typed into that field.

69

00:05:39,320  -->  00:05:46,100
So if this was a search bar like this is or if it was a user name or password field you type in any

70

00:05:46,110  -->  00:05:47,970
text you want into it.

71

00:05:47,960  -->  00:05:53,150
So in a lot of HVM all forms I would have to find the submit button and then get an element object for

72

00:05:53,150  -->  00:05:55,600
that and then call it click method.

73

00:05:55,790  -->  00:05:59,350
But selenium helps us out with a team of forms.

74

00:05:59,390  -->  00:06:06,560
I can just take this search fields form and call submit and slummy will automatically find the HMO form

75

00:06:06,560  -->  00:06:13,760
associated with this text field and then invoke it submit action so I don't actually have to click on

76

00:06:13,760  -->  00:06:17,210
anything.

77

00:06:21,260  -->  00:06:28,620
And you can see I've just done a search for Zuo fee which is the name of my cat who features prominently

78

00:06:28,620  -->  00:06:36,790
in all of these Chapter 17 pictures.

79

00:06:37,500  -->  00:06:42,210
So we've been interacting with the web pages in this browser but using the browser objects method you

80

00:06:42,200  -->  00:06:44,900
can also control the browser itself.

81

00:06:45,120  -->  00:06:47,640
I wanted to press the back button on the browser.

82

00:06:47,730  -->  00:06:50,820
I could just call the back function.

83

00:06:50,880  -->  00:06:53,310
Same thing with a forward.

84

00:06:54,040  -->  00:07:00,770
Or if I wanted to hit the refresh or reload button I could just call the Refresh method and once I'm

85

00:07:00,770  -->  00:07:07,520
done with everything and I want to close the browser I can just call the quit method and that will immediately

86

00:07:07,520  -->  00:07:11,310
make the browser that selenium launched disappear.

87

00:07:11,750  -->  00:07:16,530
So we've seen how selenium can be used to interact with the browsers web pages and interact with the

88

00:07:16,520  -->  00:07:18,520
browser itself.

89

00:07:18,620  -->  00:07:20,930
Let's just bring up a nother window

90

00:07:29,490  -->  00:07:34,820
now and take a look at how your Python scripts can use selenium to read the content of the web pages

91

00:07:34,820  -->  00:07:35,660
.

92

00:07:36,500  -->  00:07:37,690
First we'll have to get in.

93

00:07:37,690  -->  00:07:43,010
Element object for the part of the web page that we want to read and say Let's just grab this one right

94

00:07:43,010  -->  00:07:43,400
here.

95

00:07:43,420  -->  00:07:44,750
We're going to inspect element

96

00:07:47,510  -->  00:07:48,740
copy unique selector

97

00:07:56,990  -->  00:07:57,270
what

98

00:08:00,250  -->  00:08:10,980
and all web elements have a text member variable that contains a string of the text inside of that element

99

00:08:10,980  -->  00:08:13,890
.

100

00:08:15,800  -->  00:08:19,920
If you want the entire text for the Web site.

101

00:08:20,810  -->  00:08:27,560
The easiest thing to do is probably just grab the H.T. mail or body element which should contain the

102

00:08:27,560  -->  00:08:33,720
entire web page.

103

00:08:34,760  -->  00:08:40,440
If we take a look at the source by pressing control you you can see that the h l element is the first

104

00:08:40,430  -->  00:08:46,900
element for the entire web page that'll contain everything in the web page.

105

00:08:48,330  -->  00:08:54,140
So selenium can do much more beyond the functions described here but this is the basics of it going

106

00:08:54,140  -->  00:09:00,420
to Web sites finding elements that we want to click on or typing text into fields and submitting forms

107

00:09:00,410  -->  00:09:00,690
.

108

00:09:00,750  -->  00:09:02,740
That's what most web browsing is.

109

00:09:02,780  -->  00:09:08,160
And now we have a way to write Python scripts that can do all of these actions automatically but if

110

00:09:08,150  -->  00:09:12,770
you want to learn more about selenium you can read the full documentation ahead.

111

00:09:12,840  -->  00:09:15,830
Selenium Desch Python.

112

00:09:16,400  -->  00:09:24,180
Read the docs dot org or read the rest of Chapter 11 of automake the boring stuff with Python.

113

00:09:24,350  -->  00:09:28,450
And to recap Tim poort selenium you need to run from selenium.

114

00:09:28,500  -->  00:09:35,750
Import web driver just running import selenium doesn't work to open the browser run web driver.

115

00:09:35,760  -->  00:09:36,650
Does Firefox.

116

00:09:36,650  -->  00:09:42,290
This will actually launch the Firefox web browser and you'll see it appear on your screen.

117

00:09:42,300  -->  00:09:48,180
The browsers get method can be used to send it to a particular Web site and the find element or find

118

00:09:48,170  -->  00:09:54,860
elements by CSSA selector method is the main way that you'll use to grab elemental objects representing

119

00:09:55,160  -->  00:09:57,360
parts of the web page.

120

00:09:57,600  -->  00:10:02,640
Once you have these element objects you can look at its text member variable to look at the HD M-L in

121

00:10:02,630  -->  00:10:09,450
that element or you can call it click or send keys method to click or type into that element and then

122

00:10:09,440  -->  00:10:15,350
the submit method will simulate clicking on the submit button for an HMO form and the browser itself

123

00:10:15,360  -->  00:10:20,370
can be controlled with the back forward refresh and quit methods.

124

00:10:20,450  -->  00:10:25,740
These will simulate the back forward and refresh buttons and also closing the browser