1 00:00:03,960 --> 00:00:09,100 Alright, so let's have a look at deleting a slice of items from a list. 2 00:00:09,100 --> 00:00:11,799 So have a look at the lists_intro file from the last lecture, that 3 00:00:11,799 --> 00:00:16,119 I've got on screen. On line 12, we replace a section of the list with the 4 00:00:16,119 --> 00:00:21,190 items in another list. We're replacing the items from position 3 to the end of 5 00:00:21,190 --> 00:00:25,570 the list, and we're replacing them with the contents of an iterable. Now instead 6 00:00:25,570 --> 00:00:29,740 of replacing them, we could delete them. To see that in action, I'm going to 7 00:00:29,740 --> 00:00:38,360 create a new Python file called outliers.py. 8 00:00:41,280 --> 00:00:43,239 Now for this example, let's assume we 9 00:00:43,239 --> 00:00:46,629 have a set of data from some experiment. We analyze the data and we get strange 10 00:00:46,629 --> 00:00:50,319 results. On further investigation, it turns out that two of our measuring 11 00:00:50,319 --> 00:00:53,890 devices are faulty. They return results that are outside the 12 00:00:53,890 --> 00:00:58,329 range of possible values, for whatever they're measuring. So we may have some data 13 00:00:58,329 --> 00:01:00,879 that's been sorted. I'm just going to paste this in, but you can pause the 14 00:01:00,880 --> 00:01:04,720 video and type it if need be. It's something like this: 15 00:01:04,720 --> 00:01:08,160 notice that I've split that over two lines, just so it fits easier on the screen. 16 00:01:08,160 --> 00:01:14,060 You don't need to do that. So the first two values are much lower than the others, 17 00:01:14,060 --> 00:01:17,500 and we can remove them from the list. So to do that, on line four I'm 18 00:01:17,500 --> 00:01:23,950 going to type del data, left and right square brackets 0:2, and let's then 19 00:01:23,950 --> 00:01:29,320 print the list after having done that, on line 5 with print data in parentheses. 20 00:01:29,320 --> 00:01:36,320 Let's run the program, and we can see that it's deleted those first two entries. 21 00:01:36,320 --> 00:01:41,380 Now the last two look a bit dodgy as well. The first one is at position 16, 22 00:01:41,380 --> 00:01:46,660 so we might try to remove them with something like this: on line 6, del space 23 00:01:46,660 --> 00:01:54,310 data, and in square brackets, 16: and let's then print out the value after 24 00:01:54,310 --> 00:01:59,490 doing that, or print out the list after doing that. Let's run the program, 25 00:01:59,520 --> 00:02:04,520 and you can see that didn't work. You may have spotted the problem. 26 00:02:04,520 --> 00:02:12,700 On line 2, the entry 350 is at position 16, but we've already deleted two items on line 4. 27 00:02:12,700 --> 00:02:16,860 So that means that 350 is now position 14, so that's why line 6 28 00:02:16,870 --> 00:02:20,830 didn't do what we wanted. Now that's pretty obvious here. We're deleting the 29 00:02:20,830 --> 00:02:24,820 items manually and you probably wouldn't make the mistake that I made on line 6. 30 00:02:24,820 --> 00:02:33,200 That should be 14, rather than 16. So if you run the program, now it works. 31 00:02:33,200 --> 00:02:36,430 Alright, so why have I spent time making an obvious mistake. I've done 32 00:02:36,430 --> 00:02:40,480 it because it's something that causes problems when you come to automate the process. 33 00:02:40,480 --> 00:02:45,340 Alright, so what I'm going to do is comment out those lines, and let's get 34 00:02:45,340 --> 00:02:49,300 the computer to delete those rogue values automatically. Now I'm going to 35 00:02:49,300 --> 00:02:52,900 assume that we've already analyzed the data and decided the correct values 36 00:02:52,900 --> 00:02:57,070 should be in the range 100 and 200, inclusive. So I'm going to use two 37 00:02:57,070 --> 00:03:01,420 variables to store those two values. So let's go ahead and do that. 38 00:03:01,420 --> 00:03:11,160 We'll say min_valid equals 100, max_valid is equal to 200. 39 00:03:11,160 --> 00:03:15,819 Now you might use a for loop to remove the outlying values, and that's where 40 00:03:15,820 --> 00:03:19,380 you'll get a problem, and the cause isn't obvious until you've seen it. 41 00:03:19,380 --> 00:03:27,580 So let's type some code in, so for index comma value in enumerate and in 42 00:03:27,580 --> 00:03:32,830 parentheses, data, and a colon outside of the parentheses. We'll have an if on the 43 00:03:32,830 --> 00:03:40,380 next line, if parentheses value is less than min_valid, 44 00:03:40,380 --> 00:03:47,920 right parenthesis, or, then parenthesis again, value is greater than max_valid 45 00:03:47,920 --> 00:03:55,460 and then parenthesis colon del data, with index in between the square brackets. 46 00:03:55,460 --> 00:04:00,740 Then moving on down to line 16, you could print out the value of data. 47 00:04:00,740 --> 00:04:06,100 Now that looks like it should work. On line 13, we test to see if the value's 48 00:04:06,100 --> 00:04:11,440 less than 100 or if it's greater than 200, and if it is, we delete it from the list. 49 00:04:11,440 --> 00:04:15,970 That's the code on line 14. But it's not going to work. My deliberate mistake a 50 00:04:15,970 --> 00:04:20,880 moment ago may have helped you to work out why, but let's run it anyway to see it fail. 51 00:04:23,620 --> 00:04:26,540 You can see the first value is 5, which should have been removed. 52 00:04:26,540 --> 00:04:31,380 At the end of the list we've still got 360, as well. Now the debugger will help to 53 00:04:31,380 --> 00:04:35,600 understand what's going on here, so let's set a breakpoint on line 13, 54 00:04:35,600 --> 00:04:38,720 and then run the code in the debugger. 55 00:04:40,860 --> 00:04:45,240 Alright so index is currently zero and value is 4. That'll pass the 56 00:04:45,240 --> 00:04:51,940 conditions on line 13, so line 14 will be executed. So I'm going to step over. 57 00:04:51,940 --> 00:04:58,380 I've got to line 14, I'm going to press step over to get over. I'm going to press step over again 58 00:04:58,380 --> 00:05:02,660 Now in the debug pane, the list now starts with the value 5. You can see that 4 59 00:05:02,660 --> 00:05:09,620 has been removed. Step over once more. Now check the values of index and value. 60 00:05:09,620 --> 00:05:15,080 Index is 1 and the value's 104. 5 has been skipped because it's now at 61 00:05:15,080 --> 00:05:20,500 position zero. So I'm going to clear that breakpoint, and instead set one on line 14. 62 00:05:20,500 --> 00:05:22,730 That's quicker than stepping through all the values because we're only 63 00:05:22,730 --> 00:05:28,240 interested in what happens after a value gets deleted. Let's resume the program. 64 00:05:28,240 --> 00:05:34,900 You can see that it stopped on line 14. We've got an index of 15 and a value of 350. 65 00:05:34,900 --> 00:05:39,320 350 is about to be deleted, so I'm going to step over to do that. We're back 66 00:05:39,320 --> 00:05:42,740 on line 12, ready to get the next item from the list, and if you count the items 67 00:05:42,740 --> 00:05:47,330 in the list you'll see that 360 is at position 15. The for loop has already 68 00:05:47,330 --> 00:05:52,340 processed the item at position 15, and we'll try to get position 16 next. 69 00:05:52,340 --> 00:05:55,600 There isn't anything in that position so the loop will terminate. So we'll step over 70 00:05:55,600 --> 00:06:00,160 one more time. The loop terminates. We've jumped to line 16, as you can see there. 71 00:06:00,160 --> 00:06:04,220 Step over one more time to terminate the program, and we can obviously see the output 72 00:06:04,220 --> 00:06:09,320 in the output pane. Okay, so that's why we didn't get the results we wanted. 73 00:06:09,320 --> 00:06:13,100 The message to take away from this is to be very careful when changing the size of 74 00:06:13,100 --> 00:06:17,510 an object, that you're iterating over. If you want to replace items with different 75 00:06:17,510 --> 00:06:22,670 ones, that's normally fine, but deleting items from a list, or any other mutable 76 00:06:22,670 --> 00:06:26,450 sequence type, while iterating forwards over it will cause problems, as we've 77 00:06:26,450 --> 00:06:31,310 seen here. So for new programmers, just remember that rule. Be very careful when 78 00:06:31,310 --> 00:06:34,850 changing the size of an object that you're iterating over. Now I've got an 79 00:06:34,850 --> 00:06:38,380 additional comment for programmers from other languages, such as C or Java. 80 00:06:38,380 --> 00:06:41,660 If you don't know another language, then don't worry about this next bit. 81 00:06:41,660 --> 00:06:44,540 It probably won't make much sense and isn't something you would have thought of 82 00:06:44,540 --> 00:06:47,330 doing anyway. But for C or Java programmers, don't 83 00:06:47,330 --> 00:06:51,680 attempt to change the loop control variable in Python. In other words, you 84 00:06:51,680 --> 00:06:54,790 wouldn't want to do something like this: 85 00:06:58,030 --> 00:07:03,770 Now that code would work in C or Java and a few other languages. Python for 86 00:07:03,770 --> 00:07:06,770 loops don't work the same and that wouldn't work. Step through the code 87 00:07:06,770 --> 00:07:11,520 yourself and you'll see index being reset, each time line 12 is executed. 88 00:07:11,520 --> 00:07:17,260 So you can't mess with the flow of a for loop in Python by manipulating the loop control variable. 89 00:07:17,260 --> 00:07:21,290 Alright so I'll delete line 15. Let's finish 90 00:07:21,290 --> 00:07:24,590 the video here and in the next one, I want to look at a couple of ways to remove 91 00:07:24,590 --> 00:07:28,780 those values. See you in the next video.