1 00:00:00,500 --> 00:00:07,740 Welcome to 11:28 which roughly covers pages 163 to 165 The automate the boring stuff with Python textbook 2 00:00:07,950 --> 00:00:14,700 in previous lessons we called the eat up compile function to create regular expression objects and these 3 00:00:14,700 --> 00:00:23,520 objects had a search method for finding the first match in some text and also a find all method for 4 00:00:23,520 --> 00:00:26,540 finding every match in some text. 5 00:00:26,820 --> 00:00:31,660 So these methods are kind of like the Find feature in a word processor program. 6 00:00:32,070 --> 00:00:33,750 In fact Idol itself has one of those. 7 00:00:33,750 --> 00:00:40,070 You can hit control F and then do a find to find some text over word processor programs. 8 00:00:40,080 --> 00:00:46,140 Also have find and replace features where they not only find some text but they can replace it with 9 00:00:46,140 --> 00:00:47,900 some other piece of text. 10 00:00:47,980 --> 00:00:51,900 I will has this as well where you just type what you want to find and then what you want to replace 11 00:00:51,900 --> 00:00:53,310 it with here. 12 00:00:53,550 --> 00:00:57,090 So you can do find and replace with regular expressions as well. 13 00:00:57,090 --> 00:00:58,500 I'll give you an example. 14 00:00:59,010 --> 00:01:02,930 First let's go ahead and import the regular expression our e module. 15 00:01:03,000 --> 00:01:10,290 And I'm going to create a means regular expression object by calling compile and this will be for secret 16 00:01:10,320 --> 00:01:11,190 agent names. 17 00:01:11,190 --> 00:01:13,090 So just like Agent Mulder. 18 00:01:13,110 --> 00:01:19,470 Agent Scully that pattern of text will be the word agent and then whatever their name is so it'll just 19 00:01:19,470 --> 00:01:24,880 be some word characters one or more of them. 20 00:01:25,470 --> 00:01:31,540 So that's our agent names regular expression object. 21 00:01:32,040 --> 00:01:39,810 So we called Find all and we're searching through some string like Agent Alice gave the secret documents 22 00:01:40,560 --> 00:01:43,390 to Agent Bob. 23 00:01:44,130 --> 00:01:48,970 Is it find two occurrences of this pattern inside the string here for Agent Alice. 24 00:01:49,010 --> 00:01:53,970 So we're matching one or more word characters until we find a non word character which will be this 25 00:01:53,970 --> 00:01:55,860 space so that's why it stops it. 26 00:01:55,890 --> 00:02:01,370 Agent Alice and then the second pattern is right here for Agent Bob. 27 00:02:01,490 --> 00:02:06,990 So let's say this is a secret message and we want to classify this and redact some parts of it and we 28 00:02:06,990 --> 00:02:11,020 don't want to give out agent Alliss and Agent Barbes names. 29 00:02:11,100 --> 00:02:17,210 So we already have a regular expression object that can find their names inside of a string. 30 00:02:17,250 --> 00:02:23,760 So now we want to do a Find and Replace and we can do that with the sub method which will do a substitution 31 00:02:23,760 --> 00:02:23,790 . 32 00:02:23,790 --> 00:02:27,900 So it's sort of instead find and replace its find and substitute. 33 00:02:28,950 --> 00:02:36,420 So we'll just call the sub method of regular expression objects and this is the text that we want to 34 00:02:36,420 --> 00:02:41,740 do the substitutions in the first argument actually be what we want to replace. 35 00:02:41,780 --> 00:02:48,090 Anytime we find a match so anytime we have Agent Alice her agent PA will just substitute with the actual 36 00:02:48,090 --> 00:02:56,520 word redacted so the sub method will return this string except with the substitutions made to it. 37 00:02:56,520 --> 00:03:01,200 So that's why it says redacted gave the secret documents to redacted. 38 00:03:01,200 --> 00:03:05,350 So now agent Ellis and Agent Barbes identities are safe. 39 00:03:05,490 --> 00:03:11,790 So the sub method will not only find all the occurrences all the matches of this pattern inside the 40 00:03:11,790 --> 00:03:14,720 string that you pass it for the second argument. 41 00:03:14,730 --> 00:03:22,140 It will also substitute this string you pass for the first argument in that return String but say the 42 00:03:22,140 --> 00:03:27,930 string that you want to substitute it with you didn't actually use part of the original pattern like 43 00:03:27,930 --> 00:03:33,950 say instead of having redacted gave the secret dagger and say we want to have Agent a. 44 00:03:33,990 --> 00:03:35,580 We're not going to give the full name. 45 00:03:35,580 --> 00:03:37,360 We just want to have a little bit of it. 46 00:03:37,410 --> 00:03:43,500 So we'll just say agent a search agent Alice but nor to know that instead of putting redacted here we 47 00:03:43,500 --> 00:03:46,310 have to basically convert agent Alice to agent a. 48 00:03:46,360 --> 00:03:48,840 We have to use part of the original text. 49 00:03:48,870 --> 00:03:52,550 We can do that with a slash number syntax. 50 00:03:53,070 --> 00:03:57,420 And the numbers stand for groups in the original regular expression pattern. 51 00:03:57,420 --> 00:03:59,650 So I'll show you what I mean. 52 00:03:59,790 --> 00:04:04,590 So first we're going to have to have something that can find the first letter of the agent's name. 53 00:04:04,590 --> 00:04:10,510 So that's pretty simple I'll just slash w just some letter word character and we'll put that in the 54 00:04:10,510 --> 00:04:16,060 group and then the rest of the name will also just be slushed W-word characters. 55 00:04:16,560 --> 00:04:18,650 So we'll just use star. 56 00:04:18,660 --> 00:04:23,060 So this will be 0 or more word characters. 57 00:04:23,310 --> 00:04:28,560 So the name part that we're looking for is one word character here in group one. 58 00:04:28,560 --> 00:04:35,730 Inside these first set of parentheses followed by 0 or more other later characters Fluss or new names 59 00:04:35,730 --> 00:04:37,290 rejects value. 60 00:04:37,290 --> 00:04:42,570 Now if I just do this exact same find the whole call with that exact same string as before. 61 00:04:42,780 --> 00:04:45,860 You can see that the matches find all since this has groups. 62 00:04:45,990 --> 00:04:50,670 It's only going to return that that group not the entire match. 63 00:04:50,690 --> 00:04:52,730 But just the group. 64 00:04:52,980 --> 00:05:00,390 So this a comes from this a sense this is the entire string and then group one in that is just a right 65 00:05:00,390 --> 00:05:07,100 here and then this B comes from this match of the pattern where Group 1 is just this B BB. 66 00:05:07,230 --> 00:05:14,040 So it's copy this line the previous sub call that we made but instead of having redacted as what replaces 67 00:05:14,040 --> 00:05:22,320 the agent name you actually want to replace it with Agent and have a slash one which means use the text 68 00:05:22,310 --> 00:05:29,880 from Group one of the match for the substituted text and then we'll just add a few asterisks afterward 69 00:05:29,880 --> 00:05:30,050 . 70 00:05:30,050 --> 00:05:35,300 Or one thing to keep in mind though we should use a raw string for this so that we want to have a literal 71 00:05:35,310 --> 00:05:37,200 backslash in this string. 72 00:05:37,590 --> 00:05:44,940 So when we run this code it returns the string agent a star star star star gave the secret documents 73 00:05:44,940 --> 00:05:51,780 to Agent B star star star so star so See here's the original regular regular expression pattern or it 74 00:05:51,780 --> 00:05:56,220 matches age and a letter character followed by zero or more letter characters. 75 00:05:56,220 --> 00:06:01,670 So in this string this text is a match as well as this text. 76 00:06:01,860 --> 00:06:06,510 And that first letter character that first word character is inside of a group. 77 00:06:06,500 --> 00:06:09,620 It's group one so in are substituted string. 78 00:06:09,720 --> 00:06:16,340 We're just going to have literally the word agent and then have Slash 1 meaning inside that match whatever 79 00:06:16,350 --> 00:06:17,420 was that first letter. 80 00:06:17,430 --> 00:06:20,580 So in this match that will be this capital A. 81 00:06:20,630 --> 00:06:23,760 In this match that will be this capital B. 82 00:06:23,750 --> 00:06:29,310 We had multiple groups inside a regular expression then the first one would be slash 1. 83 00:06:29,340 --> 00:06:34,220 The second set of print the C's would be slashed to in the subdued string. 84 00:06:34,230 --> 00:06:37,430 The third group would be slash 3 and so on. 85 00:06:37,440 --> 00:06:41,850 So basically what this slash number syntax does is say in the substituted string. 86 00:06:41,850 --> 00:06:46,490 I want some part of the original matching string. 87 00:06:46,620 --> 00:06:47,840 So this is pretty powerful. 88 00:06:47,850 --> 00:06:54,350 We can now use regular expressions in a Find and Replace feature. 89 00:06:54,600 --> 00:06:58,190 Next let's talk about verbose regular expression strings. 90 00:06:58,250 --> 00:07:04,560 So the regular expression strings the string that we pass to the compile function can look kind of awkward 91 00:07:04,560 --> 00:07:08,600 especially when they get really big room with that phone number. 92 00:07:08,630 --> 00:07:13,830 Regular Expression example that we had where we would have something like slash slash the slushie for 93 00:07:13,830 --> 00:07:19,700 the area code fall by the first three numbers then another dash and then four more numbers. 94 00:07:19,830 --> 00:07:24,600 This regular version can really get out of hand and start looking really complicated and be hard to 95 00:07:24,600 --> 00:07:26,640 read when we're looking back at our code. 96 00:07:26,630 --> 00:07:30,610 Two weeks later after we wrote it and thinking like oh what what does this mean. 97 00:07:30,610 --> 00:07:36,300 It's three digits and bubble on it have comments and stuff but that's that's not going to help too much 98 00:07:36,300 --> 00:07:36,330 . 99 00:07:36,330 --> 00:07:44,400 So there's a nother format called the verbose for a format when you call the party compiles a function 100 00:07:44,420 --> 00:07:44,780 . 101 00:07:45,090 --> 00:07:50,140 Let's do our phone number example right here. 102 00:07:50,390 --> 00:07:53,960 Area code three digits first three digits last four digits. 103 00:07:54,120 --> 00:08:00,050 So in verbose mode whitespace in this string doesn't reflect the actual pattern that we want it to match 104 00:08:00,070 --> 00:08:00,270 . 105 00:08:00,450 --> 00:08:05,940 So what this means is that we can use the triple quotes here to make this a multi-line string. 106 00:08:06,020 --> 00:08:12,840 We could add new lines here and these new lines aren't going to be a part of the pattern that we're 107 00:08:12,840 --> 00:08:13,650 looking for. 108 00:08:13,760 --> 00:08:17,060 So the text doesn't have to contain new lines in order to match this pattern. 109 00:08:17,190 --> 00:08:17,980 And this is great. 110 00:08:18,000 --> 00:08:23,390 This makes it a lot more readable because we can also add comments inside of this string when we're 111 00:08:23,390 --> 00:08:27,890 using verbose mode and add a few spaces here and remember whitespace. 112 00:08:28,110 --> 00:08:33,230 All these space characters aren't going to be a part of the actual pattern itself because to add a pen 113 00:08:33,240 --> 00:08:33,810 symbol. 114 00:08:33,820 --> 00:08:39,150 Remember this is inside the string so it's technically not a Python comment but it uses the same syntax 115 00:08:39,170 --> 00:08:47,720 inside of this string and say you know which mark this as area code and a bunch of spaces here say first 116 00:08:47,730 --> 00:08:56,210 dash and so we can add comments as part of our regular expression which really helps make this a lot 117 00:08:56,220 --> 00:08:59,630 more readable later on when we're looking at this code. 118 00:08:59,630 --> 00:09:04,620 I mean say oh right I remember what all of these different parts of this giant regular expression stand 119 00:09:04,620 --> 00:09:09,170 for. 120 00:09:10,040 --> 00:09:15,720 And this is really helpful when we're coming up with really complicated patterns so say we have a group 121 00:09:15,720 --> 00:09:16,840 right here. 122 00:09:17,390 --> 00:09:22,850 We now have area code with old friends. 123 00:09:23,180 --> 00:09:24,990 We get to have this be or 124 00:09:30,480 --> 00:09:35,110 matching an area code that actually does have parentheses around the three digits. 125 00:09:35,220 --> 00:09:41,940 So we'll have to escape this opening and close parentheses have a space afterwards. 126 00:09:42,850 --> 00:09:51,250 We could just make the cut and say or area code with parentheses and no dash. 127 00:09:53,310 --> 00:10:00,510 So this helps let us add documentation to our really complicated looking regular expressions I say later 128 00:10:00,500 --> 00:10:05,610 on if we had something like we would also match an extension that comes right after this. 129 00:10:05,610 --> 00:10:13,860 We'd say OK there could also be a space followed by the letter X and maybe like two or more digits two 130 00:10:13,860 --> 00:10:22,950 to four digits for an extension number out of common saying the extension like x 1 2 3 4. 131 00:10:23,070 --> 00:10:29,310 And so we can have really complicated regular expression strings except verbose mode helps us by allowing 132 00:10:29,310 --> 00:10:35,360 us to format it by adding spaces and comments that aren't actually going to be part of the regular expression 133 00:10:35,460 --> 00:10:40,970 pattern but adding these spaces in comments really helps us make this code understandable that helps 134 00:10:40,980 --> 00:10:41,870 make it readable. 135 00:10:41,880 --> 00:10:49,010 The last thing I'm going to cover is this second argument to the compile function. 136 00:10:49,470 --> 00:10:55,820 So the last lesson I talked about how you could pass ignore case or simply capital I and this would 137 00:10:55,830 --> 00:11:00,660 make the regular expression ignore any case sensitivity issues. 138 00:11:00,680 --> 00:11:06,270 You could also pass dot all which would make the dot character literally match everything including 139 00:11:06,260 --> 00:11:07,210 the new line. 140 00:11:07,250 --> 00:11:13,580 But what if you wanted to use both ignore case and dot all you can only pass one value for this second 141 00:11:13,590 --> 00:11:14,820 argument. 142 00:11:15,000 --> 00:11:21,360 So the way that the compile function solves this is with this kind of esoteric syntax where you would 143 00:11:21,360 --> 00:11:28,200 have ignore case and then have the bitwise OR operator which is just this vertical pipe character to 144 00:11:28,190 --> 00:11:33,450 combine all of the options that you want to pass so if you want to get to the ignore case and not all 145 00:11:33,540 --> 00:11:40,280 and verbose mode you could combine each of these with a bit ys or operator. 146 00:11:40,430 --> 00:11:43,380 So bitwise operators are kind of beyond the scope of this book. 147 00:11:43,400 --> 00:11:46,390 And these aren't really used anywhere else. 148 00:11:46,400 --> 00:11:48,270 They're only used in this compile function. 149 00:11:48,260 --> 00:11:53,170 So this isn't a general Python thing that you can use with any function call that you make. 150 00:11:53,180 --> 00:11:59,270 It's really just for this second argument to the compile function and it's a syntax that was used in 151 00:11:59,370 --> 00:12:04,760 older programming languages and Python just sort of copied it because other languages at the time were 152 00:12:04,760 --> 00:12:05,240 using it. 153 00:12:05,250 --> 00:12:08,370 But it's kind of old fashioned and sort of weird now. 154 00:12:08,420 --> 00:12:14,150 But this is sort of what we're left with so if you ever want to use multiple of these options the second 155 00:12:14,390 --> 00:12:17,650 argument then just combine them with the bitwise OR operator. 156 00:12:17,660 --> 00:12:19,310 This piped character. 157 00:12:19,690 --> 00:12:26,450 So to recap the sub method will do substitutions with real regular expression objects. 158 00:12:26,490 --> 00:12:33,510 They will take some string and then any time it finds the matching pattern it will substitute that with 159 00:12:33,500 --> 00:12:38,030 some other string and return the string with all of these substitutions. 160 00:12:38,040 --> 00:12:45,570 You can also use the slash 1 slash to slash three syntax to substitute group 1 2 and 3 and so on from 161 00:12:45,570 --> 00:12:49,410 the regular expression pattern into the substituted string. 162 00:12:49,400 --> 00:12:55,340 You can also pass Arita verbose to the compile function that lets you add whitespace and comments to 163 00:12:55,350 --> 00:12:59,430 the regular expression string that you're passing to the compile function. 164 00:12:59,420 --> 00:13:05,970 This helps make the regular expression code a lot easier to read and if you want to pass multiple arguments 165 00:13:06,120 --> 00:13:13,680 such as dot all ignore case and verbose as the second argument to the compile function you can combine 166 00:13:13,670 --> 00:13:17,020 them all together with the vertical pipe bitwise OR operator