- This topic has 12 replies, 4 voices, and was last updated 16 years, 8 months ago by
pteeter.
-
AuthorPosts
-
May 5, 2008 at 9:39 pm #372582
larkost
ParticipantI was playing around a bit with InstaDMG and ran into two issues:
1) Other than duplicating the whole structure there is no real way of having more than one build train
2) There is no way for people to share the base OS updates with each other. Everyone has to figure out the order of updates themselves, and download them into the right folders.
So I created a script to try and deal with these two issues. I have got a base-line version ready, but it is still more than a little rough around the edges, and there is a lot of functionality that I still want in it. But I thought I would get it out there so that people could start playing around with it.
[b]The Basics[/b]
You just need to place the script in the same folder with the instadmg.bash script. Then run it from the command line with a list of catalog files as the arguments. The script will read through the catalog files and try to place the files referenced in the catalog into the proper places in the InstaDMG folders. If it can’t find the referenced pkg in its cache folder (which it will create) it will look in the web locations listed in the catalog and download the file, unpack it, and put the pkg into its cache.
The idea is that once someone puts together the latest version of the catalog file, everyone else can get get that from him/her and run the script and have their base image all ready for the custom packages that everyone has (which can be in a second catalog file).
I am including a sample catalog file, but this should be seen more as a format reference than an actual working model.
This first draft is going out without a specific license. I intend on putting out a version under something like the BSD license, but need to talk the specifics over with my employer first (I don’t anticipate any problems). For the moment you can use it all you want, but please don’t distribute it further (it is not ready for that yet).
[b]The Status[/b]
The script that I am releasing today is a first try at the idea. It works as long as there are absolutely no problems. If there are problems it will probably stop dead in its tracks with an unhelpful error (usually mentioning that I need better error handling). There are also a long list of features that I think are missing at this point, but I would like to get this out there to see what other people feel is missing.
At this point the script is probably not for the faint of heart, but rather for people who are willing to take a stab at things.
Also, if you are reading through the catalog file you should ignore that it says that you can use a 2 entry form. That might work, but is totally untested.
And one other thing: you have to clear out your InstaDMG folders (“CustomPKG” and “AppleUpdates”) before you start.
[b]What is Missing[/b]
Here is a partial list of what I think the script is missing, in no particular order:
– Much better error handling – I have deliberately left this for latter
– implement controlling the base image (the image of the disk), so that build trains from multiple versions of the OS are possible
– for bonus points, allow this to detect the build number of the os and do something useful with it
– A GUI – I know people are going to want this, and have structured parts of the current script so that I can layer it on top, but there is a long way to go before that makes sense.
– Handling of zip files and .tgz – there are references for zip files, but I have not yet got around to coding for them
– automatic guessing of what file inside a archive is the package (if there is one one pkg or mpkg in side)
– ability to handle archives stored on smb, ftp, or afp
– allow a catalog to import other catalog files (including ones through http) so that we can more easily piggy-back on other’s work
– handle dmg’s stored in zip or tgz
– real documentation
– a real licensee
– continue to make it more object-oriented (the first proof-of-concept was completely function-based, this one is more of a hybrid)
– allow for multiple files with the same name in the cache folder[b]Download it:[/b]
[url]http://www.stanford.edu/~larkost/InstaUpToDate0.1.zip[/url]
May 10, 2008 at 1:01 am #372641larkost
ParticipantI am a little surprised that no-one commented, but in any case, here is a second version of the script.
I am also including a copy of the instadmg.bash script that I have made a couple of modifications on (kills process that are using things in the image, and that better handles finding the pkg’s inside folders).
The big additions to this are the ability to include one catalog file in another, and the ability to have a folder that contains all of the files for a package. The latter is useful in including the XCode mpackage and its folder of packages.
I am also including my “vanilla” catalog file that generates an up-to-date 10.5.2 dmg from a 10.5.0 Retail disk with all the generic updates. The resulting output from InstaDMG should be useable on any Intel mac, except possibly the MacBook Air and the latest iMac. I am also including a catalog file that I am currently working on to get XCode working (not yet completely tested).
[url]http://www.stanford.edu/~larkost/InstaUpToDate0.2.zip[/url]
Please: feedback is appreciated!
May 15, 2008 at 11:37 pm #372774larkost
ParticipantI am now up to version 0.3. The big changes:
– A couple of bug fixes that prevented the script from working twice in a row in some cases
– For folders of installers (like XCode) where you only want a single installer to run you can put in a InstalThisOneOnly soft-link to the appropriate item
– ability to nest one .catalog file inside another (so you can create a base image, and then things based off that)
I have also included a heavily modified version of InstaDMG with this, and in fact my script should probably be run with this (definitely if you are using the InstalThisOneOnly bit). Here are some of the changes:
– Disk 1 of the OS mounts hidden, so there are less problems to run into if you are using your computer during the install. Next I will probably do the same for the target dmg.
– A lot of small changes in logging. And the ability to set how much of this should also go to the tty.
– Better ability to use folders of installers (such as XCode)
– quitting any process that has open files on the target volume before scanning
– correcting some soft-links that get mis-set by the iLifeSupport installer
– touchups in a lot of areas
I am putting the latest version in about the same place:
[url]http://www.stanford.edu/~larkost/InstaUpToDate0.3.zip[/url]
I have been using this code to successfully create images, so it should be in good working order (at least on 10.5), but there are absolutely no guarantees.
Oh, and this will probably be the last update for a couple of weeks, as I am going on vacation next week, and have to finish a presentation for SIGUCCS after that.
May 19, 2008 at 4:59 pm #372802pteeter
ParticipantSo, am setting up my catalog files now.
I work with 4-5 different image ‘classes’ and for me InstaUpToDate makes perfect sense.
I have not run the instaUpToDate.py script just yet but should be ready to give it a try within a day or so.
It really all depends on how perversely modular one wants to be in their imaging process.
I plan to rely heavily on the use of the ‘include file’ option. Hopefully that doesn’t prove to be foolish.
I’m managing 100 or so packages to perfect my image. About half will be included on every class of image, the rest vary.
larkhost – do you have the resources to test a bit with Tiger? I’m planning to test with Tiger, I’ll be sure to update you on what I find.
Agree with Joel/Josh – great work, powerful stuff.
Josh – will you make time soon to look at larhost’s mods to your instadmg.bash script?
May 19, 2008 at 11:35 pm #372810pteeter
ParticipantHere’s a question or two.
If InstaUpToDate.py parses include-file directives first in a catalog file, and the included catalog file points to another catalog file…will the python script still work?
Also, if a pointer to an included catalog file references a catalog that only has ‘Third Party Software’…will the python script still parse through the different sections of the original catalog file?
Again, haven’t tried it yet…just curious.
May 20, 2008 at 1:12 pm #372816larkost
Participantpteeter: At the moment most of my development time is in getting a 10.5 image ready, so working on “new” things for 10.4 has to take a back seat. I also am going away from 10.4 systems, so won’t really have an environement where it gets nearly the testing that 10.5 does. Mainly it is going to come down to whether the 10.4 python environment has everthing that I am using.
And thanks for helping me test!
And as to include fies: they are parsed just as they are encountered. So if you put them in the beginning of the file then they are edded before other things in that file. If you put them in at the end, then they are inserted after, and in between is in between. But all system updated come before all applicatinon, etc. If you look at the order of the sections in my sample catalog file, that is the order things come in, within a section they are then ordered by where they appeared.
And you can nest includes as deply as you want. It will throw an error if you include a file that has already been processed (poor mans looping detection).
And you can include as many sections as you want, and can even reverse the order of them, or put multiples of the same section in (they will still wind up in the same order)
May 20, 2008 at 1:34 pm #372817pteeter
ParticipantThanks for the reply.
I sort of accidentally ran the checksum.py script on my G5/Tiger xserve last night. It failed but no worries. I’ll test populating my Tiger InstaDMG environment on Leopard and see how it goes. I think it should still work.
I’m storing all of my image building materials on an XserveRAID with a folder shared out over AFP/SMB/NFS. One of those protocols ought to let me mount the share to a Leopard client for running InstaUpToDate, maybe.
I populated a sample InstaDMG environment last night via InstaUpToDate using nested catalog files. Worked great for me. The messages that display on the command line are informative. I suppose I’d like a log file to tail and to check at the end, but that’s a minor detail. I also want to collect my catalog files in a Catalog folder, I’ll try this and see if InstaUpToDate behaves.
I hope to run a Tiger populating test later today.
The more I use InstaDMG (and InstaUpToDate), the more it would help me if I could execute from a network shared folder.
Joel suggested, a week or so back, that NFS might work with InstaDMG. I haven’t had any luck myself but I didn’t investigate that deeply.
Good stuff larkhost. Your tool is going to help me a ton.
June 17, 2008 at 11:09 pm #373159larkost
ParticipantOk, another small release, and this one is just about bug-fixes (if memory serves… this was all pre-WWDC stuff):
[url]http://www.stanford.edu/~larkost/InstaUpToDate0.3a.zip[/url]
July 24, 2008 at 5:07 am #373495pteeter
ParticipantI haven’t gone thru your script yet but I’d guess we’ll need to change references to AppleUpdates over to BaseUpdates with Josh’s change to the InstaDMG directory framework.
No?
July 24, 2008 at 5:24 am #373496larkost
Participant[QUOTE][u]Quote by: pteeter[/u][p]I haven’t gone thru your script yet but I’d guess we’ll need to change references to AppleUpdates over to BaseUpdates with Josh’s change to the InstaDMG directory framework.
No?
[/p][/QUOTE]That is probably the only change that needs to be made on the InstaUpToDate side, but there are a number of changes that I made to InstaDMG in the custom version that I put out that need to be brought into the latest InstaDMG. I was actually going to start work on that tomorrow, since I need to create an image in the next couple of weeks. Now that InstaDMG is in a repository I can submit changes to it in a better fashion.
July 24, 2008 at 7:19 pm #373501pteeter
ParticipantIt seems that if I update the appleUpdatesFolder setting with the proper directory, BaseUpdates, instaUpToDate is happy enough.
Looking forward to your mods in InstaDMG.
-
AuthorPosts
- You must be logged in to reply to this topic.
Comments are closed