1 00:00:05,280 --> 00:00:10,440 so we are gonna look on a way of updating values stored in a shelve and also 2 00:00:10,440 --> 00:00:15,580 take a look at a common way to increase performance and also discussing go 3 00:00:15,580 --> 00:00:19,550 through problems that often surprise people new to the shelve module so what 4 00:00:19,550 --> 00:00:23,570 we're going to do is create a new Python file and we are gonna store something slightly more 5 00:00:23,570 --> 00:00:26,430 complex other than simple strings 6 00:00:26,430 --> 00:00:33,480 and use shelve to store recipe ingredients as a list so go ahead and create a new Python 7 00:00:33,480 --> 00:00:45,760 file for this will call this recipes and start typing so I'm gonna import shelve again.... 8 00:00:45,760 --> 00:01:03,600 and type in some so... 9 00:01:21,240 --> 00:01:45,500 and..and one more....ok I'm gonna type with...so we are gonna save it 10 00:01:45,500 --> 00:02:10,930 to a shelve called recipes and as recipes...and we should be able to do recipes... 11 00:02:20,750 --> 00:02:39,450 ...so the list of ingredients are 12 00:02:39,450 --> 00:02:42,970 simple and I've done this to cut down on the amount of typing that has to be done 13 00:02:42,970 --> 00:02:46,800 not because I can't cook anything more complicated although my partner would 14 00:02:46,800 --> 00:02:51,260 probably think that is true so after creating a list what we do is we open the 15 00:02:51,260 --> 00:02:58,230 shelve and we say for the meals to it you can see that on lines 10 through 13 so this point now I can run this 16 00:02:58,230 --> 00:03:03,380 so right click it to run it for the first time and you can see over here we got 17 00:03:03,380 --> 00:03:07,290 our database recipes.db our shelve created successfully 18 00:03:07,290 --> 00:03:11,140 once we've done that I can close the run window in comment out 19 00:03:11,140 --> 00:03:19,290 some of the code because we don't need those anymore because we got our data written and likewise I'm going to comment these out because 20 00:03:19,290 --> 00:03:22,860 they've now been written to the shelve and we don't need to write to them again 21 00:03:22,860 --> 00:03:28,720 and you can see the list itself was pretty simple we can come back and actually add recipes 22 00:03:28,720 --> 00:03:32,380 so recipes... 23 00:03:32,380 --> 00:03:46,770 ...then will just do a for.... 24 00:03:46,770 --> 00:03:55,160 ....and to look up at our shelve the value for that is passing our snack key and we should be able to run this 25 00:03:55,160 --> 00:03:58,910 and you can see that we were able to get the other values that were previously stored as well as the 26 00:03:58,910 --> 00:04:05,610 soup entry that was stored from line 14 and outputted fine on the screen again everything 27 00:04:05,610 --> 00:04:07,470 is working as expected 28 00:04:07,470 --> 00:04:11,580 so continuing on what we do now is to look at what happens when we modify one of 29 00:04:11,580 --> 00:04:15,330 the list so adding butter to the bacon lettuce and tomato sandwich is 30 00:04:15,330 --> 00:04:19,240 probably good idea and we also add some tomato to the the pasta 31 00:04:19,240 --> 00:04:25,590 dish so to do that what we do is actually what I'll do also is I'll comment out this soup because we 32 00:04:25,590 --> 00:04:33,660 written that to the database but in terms of pending items you type.... 33 00:04:33,660 --> 00:04:45,810 ...and we can do.... 34 00:04:45,810 --> 00:04:53,420 ...so we are adding tomato to our pasta and butter to our blt... 35 00:04:53,420 --> 00:04:59,750 ...so looking at that you think that would be all you need to do so we got the code in there 36 00:04:59,750 --> 00:05:01,200 you can see we are adding a .append 37 00:05:01,200 --> 00:05:06,100 it would seemingly be the case that adding butter to BLT this is how 38 00:05:06,100 --> 00:05:10,820 we go about it and we can run this program to see what happens but we look at the screen 39 00:05:10,820 --> 00:05:15,330 we can see what's happening is that it's not updated the pasta still has only got 40 00:05:15,330 --> 00:05:20,280 pasta and cheese and the BLT still only got bacon lettuce tomato and bread so there are 2 41 00:05:20,280 --> 00:05:22,230 items were dependent 42 00:05:22,230 --> 00:05:27,390 the problem here is the shelve has no way to know that these lists are changed so 43 00:05:27,390 --> 00:05:31,760 what's actually happened is that we've appended items to a copy of the list read 44 00:05:31,760 --> 00:05:35,750 into memory but we haven't provided any trigger for the shelve to write data 45 00:05:35,750 --> 00:05:40,470 back out again and this is very confusing when you first starting with shelve 46 00:05:40,470 --> 00:05:44,660 the reason that it works this way is to keep the disk access to an 47 00:05:44,660 --> 00:05:45,100 absolute minimum 48 00:05:45,100 --> 00:05:49,510 so that values aren't continually written to disk but it does have this unfortunate 49 00:05:49,510 --> 00:05:54,150 side effect that when accessing the same keys value after a change causes it to 50 00:05:54,150 --> 00:05:58,580 be read from the shelve and that's the unchanged value is returned 51 00:05:58,580 --> 00:06:03,070 rather than the value that you think it would be in this case by our tomato 52 00:06:03,070 --> 00:06:07,870 from our two list there is two ways to solve this so we're going to do is look at 53 00:06:07,870 --> 00:06:12,200 both and discuss the implications of each approach so first 54 00:06:12,200 --> 00:06:16,230 we can append to a copy of the list or do whatever immutable objects being used 55 00:06:16,230 --> 00:06:20,710 for the value and assign the copy back with the same key exactly as we did when we 56 00:06:20,710 --> 00:06:24,470 change lime to be great with tequila which we did in 57 00:06:24,470 --> 00:06:30,980 the previous video so we'll do that with a bit of code to show what I mean, so will comment that out 58 00:06:30,980 --> 00:06:38,840 and how we would do it would put temp... 59 00:06:38,840 --> 00:06:40,870 ....so we are retrieving our existing BLT 60 00:06:40,870 --> 00:06:58,180 the items in that list and we are gonna go temp....and then we do recipes.... 61 00:06:58,180 --> 00:06:59,570 ....and like wise for the paste... 62 00:07:05,190 --> 00:07:23,020 ....and then we do.....so you can see what we are doing there is creating the list 63 00:07:23,020 --> 00:07:30,440 from the recipes shelve grabbing the BLT in this case where appending the entry 64 00:07:30,440 --> 00:07:34,970 we want to append butter in this case and then using still updating the effectively in the shelve for 65 00:07:34,970 --> 00:07:39,880 BLT to be whatever the contents of the list is the temp_list is and we are doing 66 00:07:39,880 --> 00:07:46,080 exactly the same to pasta on lines 22 to 24 so if we run this we can now see that 67 00:07:46,080 --> 00:07:50,880 BLT has got butter and our pasta has got court tomato as well just as 68 00:07:50,880 --> 00:07:56,280 confirmation of that we should then be able to comment out the scope because we have updated it 69 00:07:56,280 --> 00:08:02,930 I'll comment that out and if we run this again now you can see we got the values 70 00:08:02,930 --> 00:08:07,050 butter and tomato still proving that the data has been written out to the shelve because we didn't 71 00:08:07,050 --> 00:08:11,950 to do any update this time when we run the code again so the advantage of this 72 00:08:11,950 --> 00:08:15,720 approach is that you gain the benefit of working with objects in memory with all 73 00:08:15,720 --> 00:08:19,790 the performance benefits that that provides because obviously a lot faster than if 74 00:08:19,790 --> 00:08:23,640 you continue reading and writing to disk but the disadvantage is that you gotta make 75 00:08:23,640 --> 00:08:27,590 sure that your reassign any mutable objects that have changed back to their 76 00:08:27,590 --> 00:08:32,990 keys in the shelves, as we did on this now commented out code on line 19 to 25 77 00:08:32,990 --> 00:08:36,480 so there's another way of doing it and the other approach involves opening 78 00:08:36,480 --> 00:08:40,330 the shelve with writeback set to true. so to see this working what we gonna do 79 00:08:40,330 --> 00:08:45,140 is add some croutons to our soup after turning writeback on for the shelf, on line 9. 80 00:08:45,140 --> 00:09:00,820 so we are gonna come down here and put recipes....but we also come up here, to our with, where 81 00:09:00,820 --> 00:09:10,090 its got recipes for shelve name....again don't worry about the red their 82 00:09:10,090 --> 00:09:14,110 which is meant to indicate a valid part of the statement but for some reason its coming out 83 00:09:14,110 --> 00:09:18,880 in red and we just run that to confirm that it does work you can see that our soup has now got 84 00:09:18,880 --> 00:09:28,450 croutons. And again, just to be sure, comment that code out, run it again, and again you see that our soup here has got 85 00:09:28,450 --> 00:09:35,180 croutons in it. So when you actually use writeback Python caches the object in memory and doesn't 86 00:09:35,180 --> 00:09:39,820 actually update the shelve file itself until you either close the shelve 87 00:09:39,820 --> 00:09:45,290 or use to sync method. If there's been a lot of changes, closing a shelve can take a while 88 00:09:45,290 --> 00:09:50,230 because all have to be written to disk at once. So this gives the advantage, as you saw, 89 00:09:50,230 --> 00:09:54,500 of sort of slightly simpler code because we didn't need to do all this code on 19 90 00:09:54,500 --> 00:09:58,090 to 26 because it was handled automatically. But the price of this, the 91 00:09:58,090 --> 00:10:02,370 disadvantage in other words, is heavy memory usage. and possibly much more heavier 92 00:10:02,370 --> 00:10:03,290 usage, depending 93 00:10:03,290 --> 00:10:05,460 on how many changes and the amount of data you're working with. 94 00:10:05,460 --> 00:10:10,460 And there's also, I mentioned, the sync method and that the writeback way of 95 00:10:10,460 --> 00:10:15,450 doing things doesn't actually update the shelve file until you close the shelve or used the sync method 96 00:10:15,450 --> 00:10:19,150 there's a potential problem with using that sync method that can catch people 97 00:10:19,150 --> 00:10:25,400 out so sync itself causes all entries in the cache to be written to disk but it also 98 00:10:25,400 --> 00:10:29,490 clears the cache as well so sync is called automatically when the shelve is 99 00:10:29,490 --> 00:10:33,900 closed but you can also use it anywhere anytime you want to force the data 100 00:10:33,900 --> 00:10:37,880 files to be updated. Now so far, we've appended to our lists by retrieving them from the 101 00:10:37,880 --> 00:10:42,200 shelve by key. As a general rule, that's the best way to do things; however there could be a 102 00:10:42,200 --> 00:10:45,480 situation where you may want to update list immediately after adding to the 103 00:10:45,480 --> 00:10:49,570 shelve. What we do is we'll add sync again, then we'll appended a drop of cream to the list. 104 00:10:49,570 --> 00:10:54,130 But we're gonna call the sync method after adding the soup, which probably 105 00:10:54,130 --> 00:10:59,440 think at first time you think is a reasonable thing to do and we're doing 106 00:10:59,440 --> 00:11:03,840 that so it's immediately saved to disk so the code is going to look something similar to 107 00:11:03,840 --> 00:11:09,440 this so we got our recipes I'm going to uncomment that again, actually what I'll do is I'll leave that commented out and put... 108 00:11:09,440 --> 00:11:21,340 recipes...so of course thats now 109 00:11:21,340 --> 00:11:26,190 essentially with wrtieback = true enabled is going to overwrite our previous entries for soups so in other words it's 110 00:11:26,190 --> 00:11:29,570 going to delete out the croutons and we are going to be left with the initial 111 00:11:29,570 --> 00:11:32,620 settings on line 6 so we are going to do that 112 00:11:32,620 --> 00:11:37,920 so if we do a recipes.sync you would think that's the way of doing it 113 00:11:37,920 --> 00:11:52,440 and I should be able to do soup....and of we run that soup no cream their, run again, no cream their 114 00:11:52,440 --> 00:12:00,400 and obviously take this code out still no cream on the soup so the reason for that is that these soup list object is 115 00:12:00,400 --> 00:12:05,510 stored in the cache and the sync causes the cache to be cleared so soup.append on 116 00:12:05,510 --> 00:12:08,580 line 29 and ill just uncomment those out 117 00:12:08,580 --> 00:12:16,160 so the souped.append cream on line 29 does add cream to the list but there's no 118 00:12:16,160 --> 00:12:21,000 corresponding list object in the cache anymore because we called recipes.sync 119 00:12:21,000 --> 00:12:24,530 on line 28 so when we loop through the shelves putting the lists were 120 00:12:24,530 --> 00:12:28,120 actually retrieving the lists from disk again so in other words the soup list that 121 00:12:28,120 --> 00:12:31,640 retrieved is not the same one that we've just added cream too that's 122 00:12:31,640 --> 00:12:35,470 because the sync is effectively in the wrong place so to be very careful if you 123 00:12:35,470 --> 00:12:41,230 decide to take this approach to call the sync method probably better advice is to not update 124 00:12:41,230 --> 00:12:42,480 objects in this manner 125 00:12:42,480 --> 00:12:47,160 use the method that we previously previously showed here something along those 126 00:12:47,160 --> 00:12:52,270 lines that's a better better method of ensuring its going to be updated so far we look at 127 00:12:52,270 --> 00:12:57,070 how to use the shelve module and how it behaves very much like a persistent 128 00:12:57,070 --> 00:13:01,840 dictionary so we summarize the features of shelve and have a look at the 129 00:13:01,840 --> 00:13:04,580 when it will be appropriate to use it before getting into the challenge now that we've mentioned 130 00:13:04,580 --> 00:13:08,690 that the keys must be strings but the values can be just about any Python 131 00:13:08,690 --> 00:13:13,110 objects and can be really literally just as complex as you need so dictionaries 132 00:13:13,110 --> 00:13:17,640 containing lists for example can be use as values in the shelves the value are pickled 133 00:13:17,640 --> 00:13:21,480 before being stored in the underlying data base fields so the same for pickle 134 00:13:21,480 --> 00:13:27,620 applies to shelve values because of that now this makes shelves really convenient and simple way to store 135 00:13:27,620 --> 00:13:32,220 programs data unless you have persistent storage yet continue to use the usual 136 00:13:32,220 --> 00:13:37,150 Python syntax without having to learn SQL which is Structured Query Language a way 137 00:13:37,150 --> 00:13:42,700 to access data bases so you don't really need to use the use that SQL to 138 00:13:42,700 --> 00:13:47,910 interrogate database here rather use the usual Python syntax in the last few video 139 00:13:47,910 --> 00:13:51,650 however with that said the shelve module does have some drawbacks and I 140 00:13:51,650 --> 00:13:57,020 mean it's not suitable for some applications so as a few examples for example 141 00:13:57,020 --> 00:14:01,990 because values are pickled before being stored and are unpickled by the value when it went 142 00:14:01,990 --> 00:14:08,210 back if your values are really complex this pickling and unpickling may impose significant overhead and 143 00:14:08,210 --> 00:14:11,790 affect performance so in other words slow down your application and the 144 00:14:11,790 --> 00:14:15,420 we mentioned that different systems may use different underlying database technology 145 00:14:15,420 --> 00:14:21,210 for storing the shelve so that data is platform agnostic so if an application 146 00:14:21,210 --> 00:14:24,960 is likely to be moved to a new system and must take its data starting with it then 147 00:14:24,960 --> 00:14:29,810 shelves aren't probably the best solution because they may or may not work properly or at 148 00:14:29,810 --> 00:14:34,380 all in that environment or system and of course this also leads to another 149 00:14:34,380 --> 00:14:39,490 potential issue and that's data from untrusted sources so deploying an 150 00:14:39,490 --> 00:14:43,160 application that uses shelves for essentially Data necessary for the 151 00:14:43,160 --> 00:14:48,680 program to run as such as the location of the maps from our adventure game would be deploying the shelve files 152 00:14:48,680 --> 00:14:53,500 with the program as well as the previous issue that the shelves may not be usable 153 00:14:53,500 --> 00:14:57,610 in some systems if the application is deployed over the internet and of course 154 00:14:57,610 --> 00:15:01,050 is that real possibility that the files can be tampered with and that would ultimately 155 00:15:01,050 --> 00:15:04,870 expose potentially user's to security threats 156 00:15:04,870 --> 00:15:08,960 also concurrent access can also be a problem with shelves although concurrent 157 00:15:08,960 --> 00:15:12,690 read accesses are safe if a program is writing to the shelve and no other 158 00:15:12,690 --> 00:15:18,330 programs should have it open or attempt to open it so the bottom line here is although shelves are very useful in the right 159 00:15:18,330 --> 00:15:22,830 place it may be preferable to store data in a database rather than using shelves 160 00:15:22,830 --> 00:15:26,740 now we are gonna be covering data bases later in the course and you'll be able to 161 00:15:26,740 --> 00:15:31,420 choose which persistent mechanism that best suit your needs after seeing how database work 162 00:15:31,420 --> 00:15:36,400 worked as well so that's it for this section on shelve I hope you have got 163 00:15:36,400 --> 00:15:40,300 a lot out of it and in the next video we got a challenge for you to practice using 164 00:15:40,300 --> 00:15:41,440 shelves see you in that video