WEBVTT

1
00:00:01.960 --> 00:00:07.120
so we are gonna look on a way of updating
values stored in a shelve and also

2
00:00:07.120 --> 00:00:12.269
take a look at a common way to increase
performance and also discussing go

3
00:00:12.269 --> 00:00:16.230
through problems that often surprise
people new to the shelve module so what

4
00:00:16.230 --> 00:00:20.250
we're going to do is create a new Python file
and we are gonna store something slightly more

5
00:00:20.250 --> 00:00:23.119
complex other than simple strings

6
00:00:23.119 --> 00:00:30.169
and use shelve to store recipe ingredients as a
list so go ahead and create a new Python

7
00:00:30.169 --> 00:00:42.449
file for this will call this recipes and 
start typing so I'm gonna import shelve again....

8
00:00:42.449 --> 00:01:00.289
and type in some so...

9
00:01:17.920 --> 00:01:42.189
and..and one more....ok I'm gonna type with...so we are gonna save it

10
00:01:42.189 --> 00:02:07.610
to a shelve called recipes and as recipes...and we should be able to do recipes...

11
00:02:17.430 --> 00:02:36.130
...so the list of ingredients are

12
00:02:36.130 --> 00:02:39.650
simple and I've done this to cut down on the
amount of typing that has to be done

13
00:02:39.650 --> 00:02:43.489
not because I can't cook anything more
complicated although my partner would

14
00:02:43.489 --> 00:02:47.940
probably think that is true so after
creating a list what we do is we open the

15
00:02:47.940 --> 00:02:54.910
shelve and we say for the meals to it you can see that on lines 10 through 13 
so this point now I can run this

16
00:02:54.910 --> 00:03:00.060
so right click it to run it for the
first time and you can see over here we got

17
00:03:00.060 --> 00:03:03.970
our database recipes.db our shelve created
successfully

18
00:03:03.970 --> 00:03:07.829
once we've done that I can close
the run window in comment out

19
00:03:07.829 --> 00:03:15.970
some of the code because we don't need those anymore because we
got our data written and likewise I'm going to comment these out because

20
00:03:15.970 --> 00:03:19.549
they've now been written to the shelve
and we don't need to write to them again

21
00:03:19.549 --> 00:03:25.400
and you can see the list itself was
pretty simple we can come back and actually add recipes

22
00:03:25.400 --> 00:03:29.060
so recipes...

23
00:03:29.060 --> 00:03:43.450
...then will just do a for....

24
00:03:43.450 --> 00:03:51.849
....and to look up at our shelve the value for that is passing our snack key and we should be able to run this

25
00:03:51.849 --> 00:03:55.590
and you can see that we were able to get the other values
that were previously stored as well as the

26
00:03:55.590 --> 00:04:02.299
soup entry that was stored from line 14
and outputted fine on the screen again everything

27
00:04:02.299 --> 00:04:04.150
is working as expected

28
00:04:04.150 --> 00:04:08.260
so continuing on what we do now is to look
at what happens when we modify one of

29
00:04:08.260 --> 00:04:12.010
the list so adding butter to the
bacon lettuce and tomato sandwich is

30
00:04:12.010 --> 00:04:15.920
probably good idea and we also add some
tomato to the the pasta

31
00:04:15.920 --> 00:04:22.270
dish so to do that what we do is
 actually what I'll do also is I'll comment out this soup because we

32
00:04:22.270 --> 00:04:30.340
written that to the database but in terms of  pending items
you type....

33
00:04:30.340 --> 00:04:42.490
...and we can do....

34
00:04:42.490 --> 00:04:50.100
...so we are adding tomato to our pasta and butter to our blt...

35
00:04:50.100 --> 00:04:56.430
...so looking at that you think that would be all
you need to do so we got the code in there

36
00:04:56.430 --> 00:04:57.889
you can see we are adding a .append

37
00:04:57.889 --> 00:05:02.789
it would seemingly be the case
that adding butter to BLT this is how

38
00:05:02.789 --> 00:05:07.500
we go about it and we can run this program to see
what happens but we look at the screen

39
00:05:07.500 --> 00:05:12.010
we can see what's happening is that it's
not updated the pasta still has only got

40
00:05:12.010 --> 00:05:16.960
pasta and cheese and the BLT still only got 
bacon lettuce tomato and bread so there are 2

41
00:05:16.960 --> 00:05:18.910
items were dependent

42
00:05:18.910 --> 00:05:24.070
the problem here is the shelve has no way
to know that these lists are changed so

43
00:05:24.070 --> 00:05:28.440
what's actually happened is that we've
appended items to a copy of the list read

44
00:05:28.440 --> 00:05:32.430
into memory but we haven't provided any
trigger for the shelve to write data

45
00:05:32.430 --> 00:05:37.150
back out again and this is very
confusing when you first starting with shelve

46
00:05:37.150 --> 00:05:41.340
the reason that it works this way is to
keep the disk access to an

47
00:05:41.340 --> 00:05:41.780
absolute minimum

48
00:05:41.780 --> 00:05:46.190
so that values aren't continually written to
disk but it does have this unfortunate

49
00:05:46.190 --> 00:05:50.830
side effect that when accessing the same
keys value after a change causes it to

50
00:05:50.830 --> 00:05:55.260
be read from the shelve and that's the
unchanged value is returned

51
00:05:55.260 --> 00:05:59.750
rather than the value that you
think it would be in this case by our tomato

52
00:05:59.750 --> 00:06:04.550
from our two list there is two ways to solve
this so we're going to do is look at

53
00:06:04.550 --> 00:06:08.880
both and discuss the
implications of each approach so first

54
00:06:08.880 --> 00:06:12.910
we can append to a copy of the list or
do whatever immutable objects being used

55
00:06:12.910 --> 00:06:17.390
for the value and assign the copy back
with the same key exactly as we did when we

56
00:06:17.390 --> 00:06:21.150
change lime to be great with tequila which we did in

57
00:06:21.150 --> 00:06:27.669
the previous video so we'll do that with
a bit of code to show what I mean, so will comment that out

58
00:06:27.669 --> 00:06:35.520
and how we would do it would put temp...

59
00:06:35.520 --> 00:06:37.550
....so we are retrieving our existing BLT

60
00:06:37.550 --> 00:06:54.860
the items in that list and we are gonna go temp....and then we do recipes....

61
00:06:54.860 --> 00:06:56.250
....and like wise for the paste...

62
00:07:01.870 --> 00:07:19.700
....and then we do.....so you can see what we are doing there is creating the list

63
00:07:19.700 --> 00:07:27.120
from the recipes shelve grabbing the BLT in this
case where appending the entry

64
00:07:27.120 --> 00:07:31.650
we want to append butter in this case and then using still
updating the effectively in the shelve for

65
00:07:31.650 --> 00:07:36.560
BLT to be whatever the contents of the
list is the temp_list is and we are doing

66
00:07:36.560 --> 00:07:42.760
exactly the same to pasta on lines 22 to 24 so
if we run this we can now see that

67
00:07:42.760 --> 00:07:47.560
BLT has got butter and our pasta has got
court tomato as well just as

68
00:07:47.560 --> 00:07:52.969
confirmation of that we should then be able to
comment out the scope because we have updated it

69
00:07:52.969 --> 00:07:59.619
I'll comment that out and if we run this again now you can
see we got the values

70
00:07:59.619 --> 00:08:03.739
butter and tomato still proving that the data has been written out to the shelve because we didn't

71
00:08:03.739 --> 00:08:08.639
to do any update this time when we run the
code again so the advantage of this

72
00:08:08.639 --> 00:08:12.409
approach is that you gain the benefit of
working with objects in memory with all

73
00:08:12.409 --> 00:08:16.479
the performance benefits that that
provides because obviously a lot faster than if

74
00:08:16.479 --> 00:08:20.329
you continue reading and writing to disk but
the disadvantage is that you gotta make

75
00:08:20.329 --> 00:08:24.279
sure that your reassign any immutable
objects that have changed back to their

76
00:08:24.279 --> 00:08:29.679
keys in the shelves as we did on this
now commented out code on line 19 to 25

77
00:08:29.679 --> 00:08:33.169
so there's another way of doing it and
the other approach involves opening

78
00:08:33.169 --> 00:08:37.019
the shelve which writes back set to true
so to see this working what we gonna do

79
00:08:37.019 --> 00:08:41.829
is add some croutons to our soup after turning
right back on the shelf on line 9 so

80
00:08:41.829 --> 00:08:57.509
so we are gonna come down here and put recipes....but we also come up here to our with where

81
00:08:57.509 --> 00:09:06.779
its got recipes for shelve name....again don't worry about the red their

82
00:09:06.779 --> 00:09:10.790
which is meant to indicate a valid
part of the statement but for some reason its coming out

83
00:09:10.790 --> 00:09:15.569
in red and we just run that to confirm that
it does work you can see that our soup has now got

84
00:09:15.569 --> 00:09:25.139
croutons and again just to be sure comment that code out
run it again and again you see that our soup here has got

85
00:09:25.139 --> 00:09:31.860
croutons in it so when you actually use writeback
Python caches the object in memory and doesn't

86
00:09:31.860 --> 00:09:36.500
actually update the shelve
file itself until you close the shelve

87
00:09:36.500 --> 00:09:41.970
or used to seat method so if there's been a lot of
changes closing a shelve can take a while

88
00:09:41.970 --> 00:09:46.910
because all have to be written to disk at once
so this gives the advantage as you saw

89
00:09:46.910 --> 00:09:51.180
of sort of slightly simpler code because we didn't need
to do all this code on 19

90
00:09:51.180 --> 00:09:54.779
to 26 because it was handled
automatically but the price of this

91
00:09:54.779 --> 00:09:59.059
disadvantage in other words is heavy memory
usage could possibly much more heavy

92
00:09:59.059 --> 00:09:59.970
usages depending

93
00:09:59.970 --> 00:10:07.149
on the number of changes and the amount of data you will be working with
and also I mentioned the sync method and that the right back way of

94
00:10:07.149 --> 00:10:12.139
doing things doesn't actually update the shelve file
until you close the shelve or used the sync method

95
00:10:12.139 --> 00:10:15.839
there's a potential problem with
using that sync method that can catch people

96
00:10:15.839 --> 00:10:22.089
out so sync itself causes all entries in
the cache to be written to disk but it also

97
00:10:22.089 --> 00:10:26.170
clears the cache as well so sync
is called automatically when the shelve is

98
00:10:26.170 --> 00:10:30.589
closed but you can also use it anywhere
anytime you want to force the data

99
00:10:30.589 --> 00:10:34.560
files to be updated now so far we've
appended to our lists by retrieving them from the

100
00:10:34.560 --> 00:10:38.889
shelve by key as a general that's the best way to
do things however ther could be a

101
00:10:38.889 --> 00:10:42.160
situation where you may want to update
list immediately after adding to the

102
00:10:42.160 --> 00:10:46.250
shelve what we do is will add sync again then
we'll appended a drop of cream to the list

103
00:10:46.250 --> 00:10:50.810
but we're gonna call to sync method
after adding the soup which probably

104
00:10:50.810 --> 00:10:56.129
think at first time you think is a reasonable
 thing to do and we're doing

105
00:10:56.129 --> 00:11:00.529
that so it's immediately saved to disk
so the code is going to look something similar to

106
00:11:00.529 --> 00:11:06.129
this so we got our recipes I'm going to uncomment
that again, actually what I'll do is I'll leave that commented out and put...

107
00:11:06.129 --> 00:11:18.029
recipes...so of course thats now

108
00:11:18.029 --> 00:11:22.870
essentially with wrtieback = true enabled is going to overwrite our previous
entries for soups so in other words it's

109
00:11:22.870 --> 00:11:26.250
going to delete out the croutons and we are going to be left with the initial

110
00:11:26.250 --> 00:11:29.309
settings on line 6 so we are going to do that

111
00:11:29.309 --> 00:11:34.600
so if we do a recipes.sync you would think that's the way of doing it

112
00:11:34.600 --> 00:11:49.129
and I should be able to do soup....and of we run that soup no cream their, run again, no cream their

113
00:11:49.129 --> 00:11:57.089
and obviously take this code out still no cream on the soup so the reason for that
is that these soup list object is

114
00:11:57.089 --> 00:12:02.199
stored in the cache and the sync causes
the cache to be cleared so soup.append on

115
00:12:02.199 --> 00:12:05.260
line 29 and ill just uncomment those out

116
00:12:05.260 --> 00:12:12.840
so the souped.append cream on line 29 does
add cream to the list but there's no

117
00:12:12.840 --> 00:12:17.680
corresponding list object in the cache
anymore because we called recipes.sync

118
00:12:17.680 --> 00:12:21.210
on line 28 so when we loop
through the shelves putting the lists were

119
00:12:21.210 --> 00:12:24.800
actually retrieving the lists from disk
again so in other words the soup list that

120
00:12:24.800 --> 00:12:28.320
retrieved is not the same one
that we've just added cream too that's

121
00:12:28.320 --> 00:12:32.150
because the sync is effectively in the wrong
place so to be very careful if you

122
00:12:32.150 --> 00:12:37.910
decide to take this approach to call the
sync method probably better advice is to not update

123
00:12:37.910 --> 00:12:39.160
objects in this manner

124
00:12:39.160 --> 00:12:43.840
use the method that we previously
previously showed here something along those

125
00:12:43.840 --> 00:12:48.950
lines that's a better better method of
ensuring its going to be updated so far we look at

126
00:12:48.950 --> 00:12:53.750
how to use the shelve module and how it
behaves very much like a persistent

127
00:12:53.750 --> 00:12:58.520
dictionary so we summarize the features of
shelve and have a look at the

128
00:12:58.520 --> 00:13:01.260
when it will be appropriate to use it before getting
into the challenge now that we've mentioned

129
00:13:01.260 --> 00:13:05.370
that the keys must be strings but the
values can be just about any Python

130
00:13:05.370 --> 00:13:09.790
objects and can be really literally
just as complex as you need so dictionaries

131
00:13:09.790 --> 00:13:14.320
containing lists for example can be use as
values in the shelves the value are pickled

132
00:13:14.320 --> 00:13:18.160
before being stored in the underlying
data base fields so the same for pickle

133
00:13:18.160 --> 00:13:24.300
applies to shelve values because of that now this makes shelves really
convenient and simple way to store

134
00:13:24.300 --> 00:13:28.900
programs data unless you have persistent
storage yet continue to use the usual

135
00:13:28.900 --> 00:13:33.830
Python syntax without having to learn SQL
which is Structured Query Language a way

136
00:13:33.830 --> 00:13:39.380
to access data bases so you don't really need
to use the use that SQL to

137
00:13:39.380 --> 00:13:44.590
interrogate database here rather use the usual
Python syntax in the last few video

138
00:13:44.590 --> 00:13:48.330
however with that said the shelve
module does have some drawbacks and I

139
00:13:48.330 --> 00:13:53.700
mean it's not suitable for some applications
so as a few examples for example

140
00:13:53.700 --> 00:13:58.670
because values are pickled before being
stored and are unpickled by the value when it went

141
00:13:58.670 --> 00:14:04.890
back if your values are really complex
this pickling and unpickling may impose significant overhead and

142
00:14:04.890 --> 00:14:08.470
affect performance so in other words
slow down your application and the

143
00:14:08.470 --> 00:14:12.100
we mentioned that different systems may use
different underlying database technology

144
00:14:12.100 --> 00:14:17.890
for storing the shelve so that data is
platform agnostic so if an application

145
00:14:17.890 --> 00:14:21.640
is likely to be moved to a new system
and must take its data starting with it then

146
00:14:21.640 --> 00:14:26.490
shelves aren't probably the best solution because
they may or may not work properly or at

147
00:14:26.490 --> 00:14:31.060
all in that environment or system and of
course this also leads to another

148
00:14:31.060 --> 00:14:36.170
potential issue and that's data from
untrusted sources so deploying an

149
00:14:36.170 --> 00:14:39.840
application that uses shelves for
essentially Data necessary for the

150
00:14:39.840 --> 00:14:45.360
program to run as such as the location of the maps from our adventure
game would be deploying the shelve files

151
00:14:45.360 --> 00:14:50.180
with the program as well as the previous
issue that the shelves may not be usable

152
00:14:50.180 --> 00:14:54.290
in some systems if the application is
deployed over the internet and of course

153
00:14:54.290 --> 00:14:57.730
is that real possibility that the files
can be tampered with and that would ultimately

154
00:14:57.730 --> 00:15:01.550
expose potentially user's to security
threats

155
00:15:01.550 --> 00:15:05.640
also concurrent access can also be a
problem with shelves although concurrent

156
00:15:05.640 --> 00:15:09.370
read accesses are safe if a program is
writing to the shelve and no other

157
00:15:09.370 --> 00:15:15.010
programs should have it open or attempt to
open it so the bottom line here is although shelves are very useful in the right

158
00:15:15.010 --> 00:15:19.510
place it may be preferable to store data
in a database rather than using shelves

159
00:15:19.510 --> 00:15:23.420
now we are gonna be covering data bases later
in the course and you'll be able to

160
00:15:23.420 --> 00:15:28.100
choose which persistent mechanism
that best suit your needs after seeing how database work

161
00:15:28.100 --> 00:15:33.080
worked as well so that's it
for this section on shelve I hope you have got

162
00:15:33.080 --> 00:15:36.980
a lot out of it and in the next video
we got a challenge for you to practice using

163
00:15:36.980 --> 00:15:38.120
shelves see you in that video