1
00:00:05,280 --> 00:00:10,174
Ok, let's see another example of using a 
set to remove duplicate values from data. 

2
00:00:10,174 --> 00:00:14,480
I'll demonstrate that using a list. We 
don't want to waste time typing input, 

3
00:00:14,480 --> 00:00:18,320
every time we run the program.
You've seen code, back in the Program 

4
00:00:18,320 --> 00:00:24,400
flow control in Python section, to get a list of 
values from user input. So I won't repeat it here. 

5
00:00:24,400 --> 00:00:29,600
I'll comment out those 4 lines, and we'll add 
some code to remove duplicates from a list: 

6
00:00:29,600 --> 00:00:32,640
Let's say we've been asked 
to analyse some survey data. 

7
00:00:32,640 --> 00:00:36,724
A car manufacturer wants to know what 
are the most popular colours for cars. 

8
00:00:36,724 --> 00:00:40,640
They've employed a market researcher to stand 
on various locations throughout the city, 

9
00:00:40,640 --> 00:00:45,668
and record the colours of the cars that pass.
Our data might look something like this: 

10
00:01:08,080 --> 00:01:13,040
They'd have a lot more data than that, of course, 
but there's enough there for our purposes. 

11
00:01:13,040 --> 00:01:18,000
One obvious question they might have to answer is, 
what colours did you record? We can answer that 

12
00:01:18,000 --> 00:01:34,613
by producing a set of the colours in the data:

13
00:01:42,160 --> 00:01:45,457
When we run the program:

14
00:01:45,457 --> 00:01:49,596
we get a set containing each colour. 
Sets contain unique elements, 

15
00:01:49,596 --> 00:01:54,712
so each colour only appears once.
Run the program a few times: 

16
00:01:57,360 --> 00:02:00,800
and you'll see that the colours 
appear in a different order each time. 

17
00:02:00,800 --> 00:02:05,050
Sets have no ordering remember.
If you want the list in alphabetical order, 

18
00:02:05,050 --> 00:02:10,250
you can use the sorted function:

19
00:02:17,850 --> 00:02:26,213
The sorted function will take any iterable, and produce a list. So the output, when I run the program this time, is a list. 

20
00:02:27,200 --> 00:02:29,002
There's another way we can do this, 

21
00:02:29,002 --> 00:02:32,746
which is useful if you need to keep the 
order that each colour was first spotted. 

22
00:02:32,746 --> 00:02:38,242
Remember that the colours were appearing in 
different orders, each time we ran the program. 

23
00:02:40,640 --> 00:02:45,280
We stopped that happening by sorting the set, 
but what if you need to keep the original order? 

24
00:02:45,280 --> 00:02:48,785
You might want to know which 
colour was seen first, for example. 

25
00:02:48,880 --> 00:02:53,219
It should be obvious that we can't use a 
set. Whenever you're interested in ordering, 

26
00:02:53,219 --> 00:02:57,901
sets are automatically ruled out.
Rather than writing code to iterate over the data, 

27
00:02:57,901 --> 00:03:01,438
and record each unique colour, 
there's a neat trick we can use. 

28
00:03:01,520 --> 00:03:07,120
It uses the fromkeys method of the dict class. 
We saw fromkeys, in the video on The remaining 

29
00:03:07,120 --> 00:03:11,927
dict methods, earlier in this section.
Dictionary keys are also unique, 

30
00:03:11,927 --> 00:03:18,583
and since Python 3.6, their ordering is preserved. 
Let's see the code, then I'll explain it: 

31
00:03:43,120 --> 00:03:47,070
Before I run it, have a look 
at the data – on line 13. 

32
00:03:47,070 --> 00:03:51,265
The first car spotted was blue, so 
blue should appear first in our output. 

33
00:03:51,360 --> 00:03:55,223
Next, we got a red car.
The one after that was blue, 

34
00:03:55,223 --> 00:03:59,338
so we don't record that one again.
The green one should be third in the output, 

35
00:03:59,338 --> 00:04:03,383
then we skip the red and blue cars – 
those colours have already been recorded. 

36
00:04:03,440 --> 00:04:07,144
So we finish with white.
Run the program, 

37
00:04:09,942 --> 00:04:15,357
and we get "blue", "red", "green" and "white" 
in the output – which is exactly what we wanted. 

38
00:04:16,240 --> 00:04:19,680
We've removed the duplicates from the 
data, and retained the order that each 

39
00:04:19,680 --> 00:04:25,680
unique value appeared, in just one line of code.
That's pretty impressive. When programmers talk 

40
00:04:25,680 --> 00:04:30,560
about Python being a very expressive language, 
this is the sort of thing they're referring to. 

41
00:04:30,560 --> 00:04:35,120
It's also one reason why Python is such 
a popular language for data analysis. 

42
00:04:35,120 --> 00:04:39,680
Ok, how does it work?
As we saw earlier, the fromkeys method 

43
00:04:39,680 --> 00:04:46,244
creates a dictionary. It gets the keys from the 
iterable we pass to it – our data, in this case. 

44
00:04:46,244 --> 00:04:51,360
When a value already appears in the dictionary, 
the new one replaces the existing key.  

45
00:04:51,360 --> 00:04:56,142
But the insertion order is preserved.
We then convert the dictionary to a list.

46
00:04:56,142 --> 00:05:00,320
That uses the keys from the dictionary.
If you're not sure what something like this is 

47
00:05:00,320 --> 00:05:06,560
doing, print it out. That's a great technique 
for understanding code, and also for debugging: 

48
00:05:06,560 --> 00:05:09,467
I'll add the code to do that:

49
00:05:24,000 --> 00:05:27,200
Run the program:

50
00:05:27,200 --> 00:05:32,160
In the last line of the output, we can see the 
dictionary that was produced by calling fromkeys. 

51
00:05:32,160 --> 00:05:36,560
The keys are in the order that they first 
appeared in the data, which is what we want. 

52
00:05:36,560 --> 00:05:42,240
When we convert the dictionary to a list, only 
the keys are used. The values are ignored. 

53
00:05:42,240 --> 00:05:45,360
We end up with a list containing 
the unique values from our data, 

54
00:05:45,360 --> 00:05:51,920
in the order that each value first appeared.
Ok, that was an interesting diversion. Hopefully, 

55
00:05:51,920 --> 00:05:57,040
you'll find that technique useful, at some point.
There's not much else to say about adding items 

56
00:05:57,040 --> 00:06:01,181
to a set, using the add method.
You pass in the item to add, 

57
00:06:01,181 --> 00:06:06,480
and it gets added to the set. If the item 
already exists, it won't be added again. 

58
00:06:06,480 --> 00:06:09,760
The add method isn't the only 
way to put items into a set. 

59
00:06:09,760 --> 00:06:14,720
There are also some set operations that update 
the contents of a set. Before we look at them, 

60
00:06:14,720 --> 00:06:19,840
we'll see some ways to delete items from a set.
See you in the next video.