1 00:00:05,280 --> 00:00:09,040 so going back looking at our code again if we opened up song.py and I just closed down 2 00:00:09,040 --> 00:00:15,080 the run window incidentally I just want to point out that the line on 132 the 3 00:00:15,080 --> 00:00:19,580 print that format doesn't work with Python 2 so in another words Python 2 doesn't 4 00:00:19,580 --> 00:00:24,130 allow printing in that way but you can import the print function from the 5 00:00:24,130 --> 00:00:28,350 future module if you wanted to work again just for Python 2 so if your running Python 6 00:00:28,350 --> 00:00:32,990 3 you don't need to do this but I'm just going to quickly show how to that fi you happen to be running Python 2 7 00:00:32,990 --> 00:00:36,130 just come up here onto the first line and add an import 8 00:00:37,910 --> 00:00:52,950 .... 9 00:00:52,950 --> 00:00:57,660 intelligence saying are you using something from an older version of Python so click on no 10 00:00:57,660 --> 00:00:59,060 for now 11 00:00:59,060 --> 00:01:02,230 so from __future__ 12 00:01:02,230 --> 00:01:09,930 import....so if you added that to your Python code and added 2 blank 13 00:01:09,930 --> 00:01:13,570 lines for Python 2 you find that this should work and that print statement that 14 00:01:13,570 --> 00:01:20,040 I pointed out should work on Python 2 but I'm gonna delete that now because of course we're dealing with Python 3 15 00:01:20,040 --> 00:01:24,980 so we ran it and then we had a quick look at albums.txt and check.file.txt 16 00:01:24,980 --> 00:01:30,930 in the previous video and they seem to be the same so what we can do as I alluded to 17 00:01:30,930 --> 00:01:33,530 we can compare 2 files in IntelliJ 18 00:01:33,530 --> 00:01:38,100 so the way to compare two files you just select them so albums.txt and holding 19 00:01:38,100 --> 00:01:42,780 command and clicking check file and holding control if you're on Windows or 20 00:01:42,780 --> 00:01:47,690 Mac and once you've selected both here on the left hand side you can come up here to the view 21 00:01:47,690 --> 00:01:54,890 menu and click on compare files or could have done a command D or a control d on Windows or Linux so 22 00:01:54,890 --> 00:02:01,830 click on compare files and its open up a window and has put contents are identical you can see 23 00:02:01,830 --> 00:02:04,620 that's actually compared with 24 00:02:04,620 --> 00:02:08,460 or compared both files and incidentally just going back to the code and if 25 00:02:08,460 --> 00:02:11,990 you select one file click one like that 26 00:02:11,990 --> 00:02:17,440 you can then go to view and click on compare with the second file you can actually grab 27 00:02:17,440 --> 00:02:20,970 that file literally from another project if you wanted to compare it that 28 00:02:20,970 --> 00:02:25,100 way so that's really useful if your comparing files that are in different projects such as your own 29 00:02:25,100 --> 00:02:29,020 code compared to the code downloaded from the resources for this course for 30 00:02:29,020 --> 00:02:33,630 example so you don't have to have the 2 files comparing within the same file but just having 31 00:02:33,630 --> 00:02:38,220 a look at that the compared results their again looking at this it's probably a 32 00:02:38,220 --> 00:02:41,780 good indication that the program is reading the data correctly but it 33 00:02:41,780 --> 00:02:46,540 doesn't prove at the moment and the program doesn't have any errors and as 34 00:02:46,540 --> 00:02:50,830 it turns out there's actually quite a serious flaw in the program but the test 35 00:02:50,830 --> 00:02:55,000 data doesn't reveal it and it's very important to test program with a range 36 00:02:55,000 --> 00:02:59,830 of different inputs and be very suspicious of data that is to neat such as the 37 00:02:59,830 --> 00:03:08,980 example take albums.txt file so we just close this comparison down and go back to the albums.txt 38 00:03:08,980 --> 00:03:13,100 examining that files it's nicely ordered with the album for each artist following on 39 00:03:13,100 --> 00:03:17,230 from each other and all the songs for an album group together but real world data 40 00:03:17,230 --> 00:03:21,880 often and probably usually isn't like that and so we should at least probably scramble the 41 00:03:21,880 --> 00:03:25,830 file up a bit to be able to test this properly now it doesn't take much 42 00:03:25,830 --> 00:03:29,330 scrambling to reveal the errors so we are gonna edit albums.txt 43 00:03:29,330 --> 00:03:34,330 and what i'm gonna do is cut out all the songs for zz top albums so come down here to 44 00:03:34,330 --> 00:03:46,420 zz top right down the bottom so what I'm gonna do is I'm going to cut out all the songs zz top album antenna 45 00:03:46,420 --> 00:03:54,790 this line 657 down through 668 so cut those command x on a Mac and 46 00:03:54,790 --> 00:04:02,070 make sure I delete the extra line as well be control X on Windows or PC and what I'm gonna do 47 00:04:02,070 --> 00:04:07,170 is I'm going to actually put them in a different spot in the file so go back up back 48 00:04:07,170 --> 00:04:12,360 up to the top and put them somewhere completely different say aerosmith will do so 49 00:04:12,360 --> 00:04:13,420 let's put it 50 00:04:13,420 --> 00:04:20,070 somewhere around here so maybe between permanent vacation and jim so just 51 00:04:20,070 --> 00:04:26,400 here lets just put it in their so we are putting it in a different order now just to see what happens if we try and run this 52 00:04:26,400 --> 00:04:30,160 and again I made sure that there's no blank lines either the lines I've 53 00:04:30,160 --> 00:04:36,090 added here or back where I cut it out which was on lines 650 making sure 54 00:04:36,090 --> 00:04:40,480 that there is no other lines there and there is a blank line on the line at 55 00:04:40,480 --> 00:04:44,970 the end of line 685 which I have left I haven't touched that so we've now split 56 00:04:44,970 --> 00:04:50,040 two artists albums up a little bit remembering that last time when we ran this 57 00:04:50,040 --> 00:04:55,590 it resulted in twenty-eight artists so lets run this again and we got a different 58 00:04:55,590 --> 00:04:58,930 result as you can see here we've now got their are 30 artist and again 59 00:04:58,930 --> 00:05:03,910 remembering those 28 last time when we ran this in the previous video and 60 00:05:03,910 --> 00:05:07,100 we just go and do that comparison again 61 00:05:07,100 --> 00:05:12,220 so albums.txt and check file selecting them both again and click on view and compare 62 00:05:12,220 --> 00:05:17,910 files and just open that compare files results but it still says in the top left hand corner 63 00:05:17,910 --> 00:05:21,840 contents are identical but its now reporting 64 00:05:22,390 --> 00:05:26,390 their is 30 artists so obviously somethings is badly wrong here because we only had 65 00:05:26,390 --> 00:05:30,790 twenty eight artists in the previously video we've now got 30 but yet the output is still the 66 00:05:30,790 --> 00:05:36,090 same so the point that we're making here is really does illustrate how important 67 00:05:36,090 --> 00:05:42,200 it is when you testing to test with varied data sets ok so that is the problem so onto 68 00:05:42,200 --> 00:05:45,180 what or how to go about fixing it 69 00:05:45,180 --> 00:05:50,590 problem is that each artists boundary in the data causes a new artist object to be 70 00:05:50,590 --> 00:05:55,070 created without considering whether there's already an object for that artists 71 00:05:55,070 --> 00:05:59,090 so as a result we end up with two objects for Aerosmith and to for zz top 72 00:05:59,090 --> 00:06:02,510 as it turns out the code to fix this is quite easy 73 00:06:03,240 --> 00:06:06,890 you really just have to check to see it is already an artist in the list before 74 00:06:06,890 --> 00:06:11,160 creating a new artist object and to do the same each time a new albums found 75 00:06:11,780 --> 00:06:15,490 but rather than fixing this version though we're going to go ahead and use a 76 00:06:15,490 --> 00:06:17,070 slightly different approach 77 00:06:17,070 --> 00:06:20,260 now the way that this program currently works does have some 78 00:06:20,260 --> 00:06:24,090 disadvantages including the need to check if there's an outstanding artist 79 00:06:24,090 --> 00:06:31,600 or album record at the end of the this is the code down here on line 116 onwards 80 00:06:31,600 --> 00:06:35,700 so we have to do that final check for the last line so the method uses 81 00:06:35,700 --> 00:06:40,070 easy-to-understand and sometimes it is necessary to do things this way as an 82 00:06:40,070 --> 00:06:43,150 example with restoring the data in a data base and we might want to ensure that 83 00:06:43,150 --> 00:06:48,530 an artist had at least one album before adding them to the database now with some data such as 84 00:06:48,530 --> 00:06:53,210 financial records it may also be a requirement to wrap the database 85 00:06:53,210 --> 00:06:58,190 insertions into a transaction so that either all the data of an artists including 86 00:06:58,190 --> 00:07:02,380 albums is written or not if it is so that way there's no chance of incomplete data 87 00:07:02,380 --> 00:07:06,530 being stored in the database similarly you may need to make sure that 88 00:07:06,530 --> 00:07:10,650 all the song for an album are added when the album is added or abandoned the attempts 89 00:07:10,650 --> 00:07:15,600 to store the data altogether so taking this approach of not adding the object to their lists 90 00:07:15,600 --> 00:07:19,350 until the related data has been read from a file is one way to sort of 91 00:07:19,350 --> 00:07:23,540 achieve this that doesn't really apply to this particular example though because 92 00:07:23,540 --> 00:07:27,220 all the data will fit in memory but with a data file that was to big for available 93 00:07:27,220 --> 00:07:31,280 memory would really be necessary to store the records as soon as everything 94 00:07:31,280 --> 00:07:35,530 related to an object had been read so although the algorithm that we've used may 95 00:07:35,530 --> 00:07:39,650 not be the best for this particular case it can be useful in other cases and that's 96 00:07:39,650 --> 00:07:45,460 why that we've actually included it here and algorithm is a series of steps 97 00:07:45,460 --> 00:07:48,870 for solving problems so I've described the algorithm for this approach briefly 98 00:07:48,870 --> 00:07:54,120 before writing the load_data function but I'm gonna bring up a flow chart to 99 00:07:54,120 --> 00:07:59,390 show this little bit more formerly and I'll just bring it up on the screen their now you can see I've got flow chart here 100 00:07:59,390 --> 00:08:04,750 so this is really describing what's what's really happening now with the 101 00:08:04,750 --> 00:08:08,750 flow of the program code and it can be useful if you haven't seen these before 102 00:08:08,750 --> 00:08:13,370 to investigate and find out more about flow charting when your trying to 103 00:08:13,370 --> 00:08:18,200 put together how something works it can be useful to put this together but you can see what 104 00:08:18,200 --> 00:08:22,650 is happening here is that we're starting the process it's reading some data it is 105 00:08:22,650 --> 00:08:24,840 then splitting the data into the various fields 106 00:08:24,840 --> 00:08:30,310 and you've seen that before and we start with a comparison here is the artist the same as the field 107 00:08:30,310 --> 00:08:36,210 field being what we have read in if it is the same then it flows over to here we do another check is the album the same as the 108 00:08:36,210 --> 00:08:37,270 field 109 00:08:37,270 --> 00:08:42,059 if it is we are going to add song to album checking to see if there's any more data if their is it goes back and 110 00:08:42,059 --> 00:08:45,950 sort of reloops through and in the case of their wasn't anymore data will save the 111 00:08:45,950 --> 00:08:49,810 artists and then we stop bullets so that is in that scenario their and that is with the 112 00:08:49,810 --> 00:08:54,490 artists was the same as the field but if the artist isn't the same as the field then we 113 00:08:54,490 --> 00:09:00,520 save the artists and we created that artist object and we move to here and we go through this decision 114 00:09:00,520 --> 00:09:04,300 again to see to check whether the album's the same as the field it is 115 00:09:04,300 --> 00:09:07,710 it comes over to hear then adds the song to the album object and sort of flows 116 00:09:07,710 --> 00:09:11,130 though their that we have talked about before that sort of process to be reading more 117 00:09:11,130 --> 00:09:15,630 lines of data or we reach the end to save the artist and stop but in the case of the 118 00:09:15,630 --> 00:09:19,540 album not being the same as the field we save the album and the artist 119 00:09:19,540 --> 00:09:25,240 object we created the album object then come up back up to here and then we 120 00:09:25,240 --> 00:09:29,110 add the song to the album object and go through to this point again so basically 121 00:09:29,110 --> 00:09:33,630 artists objects isn't saved until the new artists is found in the data set or 122 00:09:33,630 --> 00:09:37,770 when their no more data to read now as I mention this could be a reasonable approach to 123 00:09:37,770 --> 00:09:42,610 take in some circumstances but I'm gonna change things so that new objects are 124 00:09:42,610 --> 00:09:47,480 save as soon as they created so we are not using a database here so by save what I mean is 125 00:09:47,480 --> 00:09:52,020 their stored in their respective lists so the flowchart of that is going to be a bit 126 00:09:52,020 --> 00:09:59,430 different so let me bring them up on the screen so close that one down here is the 2nd 127 00:09:59,430 --> 00:10:03,210 flow chart so the main difference with this one and I'm not gonna go through it all then 128 00:10:03,210 --> 00:10:07,180 now but you can sort of go through that at your leisure but outlining the major difference and 129 00:10:07,180 --> 00:10:12,000 that's that the objects are save as soon as they created and that means that an 130 00:10:12,000 --> 00:10:16,670 artist we put into the list before any albums are added to it likewise an album 131 00:10:16,670 --> 00:10:20,450 object is going to be added to the artists list of albums before any songs have been 132 00:10:20,450 --> 00:10:25,310 added to it now if we were using a database to store these details what 133 00:10:25,310 --> 00:10:29,450 that means is that if something goes wrong we could end up with albums in a 134 00:10:29,450 --> 00:10:33,890 database for with some songs missing but for this example though you know when where 135 00:10:33,890 --> 00:10:36,830 we're reading everything into memory the code using the new 136 00:10:36,830 --> 00:10:40,850 algorithm ends up being a lot simpler so I'm gonna add some extra steps to check 137 00:10:40,850 --> 00:10:45,550 if an artist or album already exists and if they do then the existing objects can 138 00:10:45,550 --> 00:10:50,380 be retrieved from the list instead of creating a new one so to do that 139 00:10:50,380 --> 00:10:55,300 lets introduce a slightly more complicated version so lets bring up the flow chart for that so I'll 140 00:10:55,300 --> 00:11:01,080 close this one down and lets open up the third flow chart and you can see the new 141 00:11:01,080 --> 00:11:04,990 sequence here to check to see whether the artists already been stored if it has 142 00:11:04,990 --> 00:11:08,730 only been stored its gonna retrieved the artists and flow on and 143 00:11:08,730 --> 00:11:12,710 likewise when we tested for the album is the album already stored if yes we are gonna retrieve the 144 00:11:12,710 --> 00:11:15,800 album's first and then move on to that if it hasn't of course we are going to 145 00:11:15,800 --> 00:11:21,510 create the album object and of course over here if the artist hadn't been stored then we create it 146 00:11:21,510 --> 00:11:25,810 which we need for the first time one thing that probably stands out looking 147 00:11:25,810 --> 00:11:29,880 at this flowchart is that the new steps that have been added into the flow for 148 00:11:29,880 --> 00:11:34,600 both artist and album practically identical and this sounds good candidate 149 00:11:34,600 --> 00:11:39,560 for function so I'm gonna start by and going back to the code and looking at adding a 150 00:11:39,560 --> 00:11:44,480 find object function before load data but we will start working on that in the next 151 00:11:44,480 --> 00:11:44,740 video