Hello all,
I think I understand the Cached image and how it works, but I wish to make greater use of it and figured I would ask before wasting an hour or two doing this. Please correct any part of this that is wrong.
How it currently works:
1) InstaDMG looks for a cached image, finding none it builds the BaseOS from the install disc and the installerchoices file.
2) On completing the mounting of the new BaseOS image, instadmg checks again for CachedImage and finding none, it creates one from the newly created BaseOS image.
3) Then InstaDMG continues with Baseupdates and custom packages.
4) The next time you load up InstaDMG, it find the BaseOS CachedImage and mounts it.
5) Next it installs all the baseupdates and custom packages.
How I WANT it to work:
1) Same as above
2) Works the same as above, unless the -BUC or -BCC switch is used (see next steps)
3) Same as above unless the -BUC or -BCC switch is used:
3a) If -BUC switch is invoked then, after installing BaseUpdates, instaDMG will then create the cached image. This is done because MOST of my images use the same major updates. Why wste the same 20 minutes every time installing 10.5.4, when I can cache that. The next time InstaDMG will run, it will run as per normal, treating the BUC cached image as a normal Base OS Cached Image, but I will have removed the 10.5.4 update from my build stream and have it only include those updates I need above that, this time around.
3b) If the -BCC switch is invoked then the Cached image will be created after the entire InstaDMG process is completed. The purpose behind this is that all my images need certain basics. These are unaltered installs such as 10.54 update, itunes, iphoto and MSOffice 08. Caching these will cut my build time down drastically. Then when I need to build an image for a certain group, I can just take the cached basic and add updates, specific settings and other software.
BUC = Base plus Updates Cached image
BCC = Base plus Custom Cached image
If my understanding of how it currently works is correct then I should be able to build out my tweaks pretty easily.
On the other hand, I can short circuit the whole thing with one other piece of information.
Can I use the hash that Larkost’s python script creates to name my cached image? If so, I won’t bother redoing the script, I will just hash one of my base and baseplus images and dump them into the cached folder.
Thanks for all advice,
knowmad
Can’t you just create one build-train with your base needs and use the output file as a cache in another build-train?
First off… a bit of a clarification/correction. The process is:
1) InstaDMG looks at the dmg that is designated the “main” OS disk and asks that for it’s internal checksum (I parse the output of ‘hdiutil imageInfo
2a) If a cached image is found it opens it and skips on to step 4
2b) If no cached image is found it opens that “main” OS disk (and any other disks) makes a new target image and starts the install.
3) Once the base OS is installed it is closed off, and moved to the cached image folder (this step can be skipped if caching is disabled)
4) The cached image is now opened with a shadow file sitting on top of it to take the changes, and the installation proceeds.
I have done a bit of thinking in the same direction (as I was writing the caching code in fact), and had some ideas but also discovered that a lot of it is really hard to do right.
The main problem that I ran into in my thought exercises was that the 10.5.x upgrades are really the first thing that needs to go on top of the base image, so that would be the obvious place to put in a cache point. However, it is also something that changes on a regular basis. I have some reservations about getting too many cached images. I already think that there will be too many produced when people are playing around with creating their installerchoices file (this was what prompted me to make the command-line switch to turn off caching). Remember: every cached image is at least a couple of Gigs big (mine are 11 Gig). Start multiplying that and we get silly fast.
Oh.. and a side note: I plan on going back and compressing the base image cache files to being compressed. At the moment they are taking up too much space. I just need to have the time to check it, and am busy on other projects.
There are some issues about what things on top of that might do depending on what files are in the Base+Combo, these files are likely to change on a regular basis, and it is going to be difficult to know where to draw the line.
There is an idea out there to try and make more use of shadow files, and even to try and layer them, or use union mounts to achieve the same results. But that is going to take a lot of testing before I am going to trust it… once again time I don’t have at the moment.
Oh… and at the moment it would be difficult to use “long-form” command line switches. I vaguely want to move InstaDMG to Python (mostly to make it easier to grab error information but also to make it easier to put a GUI on top of it), and if that happens then it will be easy to use long-form command line switches.
[QUOTE][u]Quote by: hetjan[/u][p]Can’t you just create one build-train with your base needs and use the output file as a cache in another build-train?[/p][/QUOTE]
At the moment you would have to do a lot of trickery to get this to work out. Or a bit of re-writing of InstaDMG.
I do have more long-term ideas, but those are going to have to wait some time if I am going to be the one to implement them.
[QUOTE][u]Quote by: larkost[/u][p]
The main problem that I ran into in my thought exercises was that the 10.5.x upgrades are really the first thing that needs to go on top of the base image, so that would be the obvious place to put in a cache point. However, it is also something that changes on a regular basis. I have some reservations about getting too many cached images. I already think that there will be too many produced when people are playing around with creating their installerchoices file (this was what prompted me to make the command-line switch to turn off caching). Remember: every cached image is at least a couple of Gigs big (mine are 11 Gig). Start multiplying that and we get silly fast.
[/p][/QUOTE]
I understand your point but would like to point to a different scenario: Let’s say you are preparing images for deployment with a base set of software (lets say MS Office, iLife, Flip4Mac) that goes into ALL your images. Then you have a couple of other build-trains that require additional software (you fill in the blanks). Most (if not all) third party software doesn’t require a specific point release of the OS (f.x. 10.5.3) to run which makes me think that you could create a build-train chain that goes:
BaseOS
Installer Choices.xml
iLife
Flip4Mac
Office2008
iMovie710
iPhoto710
Simple enough. Now imagine we could use the OutputFile as the next build-train’s BaseOS Cache we have something. This time we could do:
OutputFile
BaseUpdates:
10.5.4
Front Row
Quicktime
etc.
CustomPKG:
CS3
iPhoto713
iMovie713
iPhoto714
iMovie714
etc
When a new Combo updater comes out we can replace the current one and save the time installing iLife (it takes a looong time), Flip4Mac and Office2008.
Ok, so
1) thank you larkost, you rock. I discovered what you did (by actually reading the script carefully instead of guessing my way through it, thanks to Sean and Pringles for their help) just seconds ago, of course long after you both wrote it and then explained it here.
2) I see your point regarding a) Space and b) Caching items of major change.
My reasoning is that I just spent how-many-months building images with 10.5.4, and how many before that with 10.5.3? I could, quite simply, have saved hours by having not had to install those major updates 20 or 30 times. I have changed venues since this has all started and honestly don’t expect to have more than one major build train, however while I am testing it out, I will build it ten or more times with tweaks. As a result, here is my new (temporary) plan.
I will now take my interim image (Base+Base Updates+Office) and (using basically your code, hope your cool with that) create a hash for it. Then dump it into the cached folder and let instadmg make the useful mistake of thinking it is the correct BaseCache.
two major questions are 1) Can I do this to a compacted DMG such as the one instadmg finally outputs? and 2) Will this work?
Thanks for taking the time to respond to all this.
knowmad
________
I think i will change my tag to ‘asking the dumb questions, so you don’t have to’
…..
maybe not
knowmad, you might be able to get that to work by naming the resulting ASR dmg according to the checksum of your base image. You will need use hdiutil to get the checksum, as it is not the SHA1 checksum, but rather the one that diskutil uses internally for the dmg.
You will have to carefully clean out your folders to make sure that you don’t get the wrong thing in there.
But I think that a better approach would be to use the same method for image creation that I use:
1) Get all my InstaUp2Date packages and catalog files updated
2) Build latest base image using InstaUp2Date + InstaDMG
3) Put the ASR image on a FireWire HD that also has a boot image on it (actually an old version of my base image)
4) Restore this onto my test machine
5) Test, and find the problems
6) Change the pkg’s on my build machine with what I think I need to change to correct the problems
7) From the test machine connect (with AFP) to my build machine and run the installers (maybe copying them locally)
8) Test, find problems, recycle to step 5 until I get things right
8a) If I feel like I need to start clean, cycle back to step 4
9) Everything looks fine (or I have to go home for the night), so cycle back to step 1
Since everything has to work with .pkg’s this means that everything is very repeatable. Sure there are some things that this is not going to work for, but it has worked for me very well. When I am really feeling good about one installer and bad about another I sometimes trigger the next build while I am still in the middle of 4-8, so that I have less to install on a cycle.
Well, Larkost, you speak sense. From a testing only perspective your system works well.
From a building several flavors of image perspective… not so good.
I think I will settle for a cross between the two.
OR
I will spend a week mucking with your code to get it to do what I want and in the end go back to the original when I have broken it beyond use.
Whichever seems more fun 😉
Seriously though, I might just create variants of the script (Command line switches are a little more than I feel like tackling) for each purpose and just use the ones I need at the time I need it…..
We shall see.
Right now I have to go do administrative paperwork so this all gets shelved until next week anyway.
knowmad