InstaUp2Date update

[larkost]

I got a question off-list and I thought I would post the answer here so that anyone else who has questions can get the answer as well. The basics of the question is about what changes have I made in InstaUp2Date to integrate it with the latest version of InstaDMG.

I am going to be referring to the version in SVN at the moment. Specifically checkin #94. I am a little fuzzy about the differences between that and 1.4b3….

The changes:

1) InstaUpToDate became InstaUp2Date, and now lives in the InstaDMG subversion repository. The latest version will always be there, and for the moment be in HEAD.

2) The first is that I am leaning heavily towards using dmg’s in InstaDMG. So for the moment I have removed the code the downloaded a dmg and pulled out a specific pkg in the dmg and used that. Instead I am relying on the idea that any dmg pulled down will only have pkg’s that you want to run. I do realize that this generalization is not universally true, but for the moment it is the generalization I am making.

I have an idea for how to allow InstaDMG to be more intelligent about this that would give the benefits of both approaches, one that would not sacrifice the potential of using just dmg’s. However I need to get some things done in the other portions of my job, so this is probably not going to be done immediately.

The new dmg type just requires 3 fields: name, location (http or local), and checksum. Examples can be found in the included catalog file (see vanilla.catalog).

3) I have added a few variables to the catalog files that allow you to set the output file and volume names for InstaDMG (I added those command line switches to InstaDMG for exactly this reason). I

4) I worked on the path handling mechanisms so that InstaUp2Date can handle its new location relative to InstaDMG (it is in AddOns). I also took the time to make sure that you can call InstaUp2Date from any folder, and it will auto-adjust it’s current working directory to run correctly.

5) I have touched up the checksum.py program to give you a line that could be used in a catalog file so you don’t have to translate it. I plan on going though and re-doing this to also handle the download of http, but that will take a back seat for a while.

6) I have introduced a -p (–process) flag that will run InstaDMG after a successful run of InstaUp2Date. If you are going to use this, then you need to run InstaUp2Date as root, so that InstaDMG will be run the proper way. I am thinking over how to do this with a bit more savvy without compromising security, but have not come up with good ideas yet.

I am happy to answer questions, but it is probably better that they be directed to this forum, as then I can answer questions only once.

[pteeter]

Good stuff larkost.

1. I / we will look for updates in the SVN repository then.

2. I’m a little fuzzy on dmgs vs. pkgs. I’ve always run checksum.py aganst pkg installers and only referred to pkg’s in my catalog files. Will that change now or not?

3. A welcome addition to both tools – output DMG file name customization. Woohoo.

4. Ok, this for me is the biggest change. The InstaDMG + InstatUp2Date dir structure now looks like this…

[code]
AddOns

—->InstaUp2Date

———>CatalogFiles

———>checksum.py

———>instaUpToDate.py

Caches

—->BaseImageCache

—->InstaUp2DateCache

instadmg.bash

InstallerFiles

—->BaseOS

—->BaseUpdates

—->CustomPKG

—->InstaUp2DatePackages

Logs

OutputFiles
[/code]

A few questions arise…

Can all .catalog files be stored in the CatalogFiles directory?
I assume the InstaUp2DateCache folder holds all DMGs and PKGs that will be linked into the BaseUpdates & CustomPKG folders?
What’s the InstaUp2DatePackages directory for now?
Which tool – instadmg or instaUp2Date – leverages the BaseOS image caching?

5. YES, one line to add to the catalog file. Nice.
6. Useful, will help with full automation when I get to doing that…if I get to doing that. javascript:emoticon(‘;)’)

Again, thanks. Your tool makes my build process far more elegant.

😉

[larkost]

[QUOTE][u]Quote by: pteeter[/u][p]
2. I’m a little fuzzy on dmgs vs. pkgs. I’ve always run checksum.py aganst pkg installers and only referred to pkg’s in my catalog files. Will that change now or not?
[/QUOTE]
Nope, you can continue to do exactly that. But take a look at the vanilla catalog file. One of the central ideas in this for me is that we can all share that file and it will take a vanilla retail disk (as in one you buy at the store) and bring it up to the latest version of everything that Software Update would add for you. When a change happens only one of us has to update the vanilla file and everyone else gets the benefits.

For the moment that person is me, but I will gladly accept submissions from others. And once we get out of the starting blocks I can see there being a collective that would contribute catalog files for things that are openly available (Firefox, AdiumX, etc). Part of the point of this is that if we can get a good structure that is widely applicable, then we have a lot of people all testing the same base image. If there is a problem someone will fix it, and we can all have more confidence in our base images, and concentrate on the things that make our installations unique.
[QUOTE]
4. Ok, this for me is the biggest change. The InstaDMG + InstatUp2Date dir structure now looks like this…

A few questions arise…

Can all .catalog files be stored in the CatalogFiles directory?
[/QUOTE]
Yep. InstaUp2Date will also search at a path you give it and I even a http: reference if you give it that.

I should also mention that I made a little change in how things work: If you give InstaUp2Date multiple catalog file it will assume that you want to setup for InstaDMG multiple times. So it will setup for one (presumably run InstaDMG), tear-down, then setup for the next run. Once for each file you give it. I am a little torn on whether I should put in logic that if it detects you have multiple files that would present an error if you don’t have the –process option selected.

I also think that there should be another option that would allow you to add a catalog file (or maybe a dmg/pkg or http reference) manually. The thought of taking in a catlog file on standard input has also gone through my head (think automated build systems). But I don’t need that yet, so it will be a while.

[QUOTE]
I assume the InstaUp2DateCache folder holds all DMGs and PKGs that will be linked into the BaseUpdates & CustomPKG folders?
[/QUOTE]
It is actually the place where InstaUp2Date will place pkg’s and dmg’s that it downloads. These should be considered disposable, as they will be re-downloaded when needed. InstaUp2Date will check here and the InstaUp2DatePackages folder to find things that will be linked in.
[QUOTE]
What’s the InstaUp2DatePackages directory for now?
[/QUOTE]
The InstallerFiles/InstaUp2DatePackages folder is intended as the place where you put the pkg’s or pkg-in-a-dmg’s that you download or create. If InstaUp2Date is the one to download the dmg then they go into Caches/InstaUp2DateCache. That way you know that you can get rid of anything in Caches without loosing data.

That reminds me.. I have to set the flag on the cache files to get them to be ignored by TimeMachine… ok… done in rev 96
[QUOTE]
Which tool – instadmg or instaUp2Date – leverages the BaseOS image caching?
[/QUOTE]
I added that to InstaDMG. It makes the 45 minutes or so of Base OS install into 6 minute process (after the first run with that image/installerchoices file). I got tired of the long wait through that while testing, so scratched that itch. There are a few other ideas like that might get implemented sometime.
[QUOTE]
Again, thanks. Your tool makes my build process far more elegant.
[/QUOTE]

I have selfish motivations in changing the code, but I am happy that others get to benefit as well.

[pteeter]

[color=Blue]Nope, you can continue to do exactly that. But take a look at the vanilla catalog file. One of the central ideas in this for me is that we can all share that file and it will take a vanilla retail disk (as in one you buy at the store) and bring it up to the latest version of everything that Software Update would add for you. When a change happens only one of us has to update the vanilla file and everyone else gets the benefits.

For the moment that person is me, but I will gladly accept submissions from others. And once we get out of the starting blocks I can see there being a collective that would contribute catalog files for things that are openly available (Firefox, AdiumX, etc). Part of the point of this is that if we can get a good structure that is widely applicable, then we have a lot of people all testing the same base image. If there is a problem someone will fix it, and we can all have more confidence in our base images, and concentrate on the things that make our installations unique.[/color]

I think what you’re aiming for with the vanilla.catalog file is admirable, and easier with Leopard too.

I believe I could fairly easily compile my Tiger-ppc and Tiger-x86 ‘vanilla’ catalog files for use like this. It was tricky at first b/c both architecture’s have separate patch groups for the OS, Safari, Java, and Airport. I’m pretty sure they are static now, save hardware specific updates. Where should I submit these for inclusion?

[color=Blue]I should also mention that I made a little change in how things work: If you give InstaUp2Date multiple catalog file it will assume that you want to setup for InstaDMG multiple times. So it will setup for one (presumably run InstaDMG), tear-down, then setup for the next run. Once for each file you give it. I am a little torn on whether I should put in logic that if it detects you have multiple files that would present an error if you don’t have the –process option selected.

I also think that there should be another option that would allow you to add a catalog file (or maybe a dmg/pkg or http reference) manually. The thought of taking in a catlog file on standard input has also gone through my head (think automated build systems). But I don’t need that yet, so it will be a while.[/color]

Both cool ideas. I have spent a good amount of time inbetween runs of InstaUpToDate (old-style) + Instamdmg cleaning up, more or less. I’ll be sure to try the multiple catalog + –process option next weekend.

[color=Blue]I added that to InstaDMG. It makes the 45 minutes or so of Base OS install into 6 minute process (after the first run with that image/installerchoices file). I got tired of the long wait through that while testing, so scratched that itch. There are a few other ideas like that might get implemented sometime.[/color]

You’re right on about the delay, a great addition. Again, I’ll try that this weekend when I build for sure.

larkost – I have to ask…are you (the creator) and me the only people using InstaUp2Date?

[larkost]

pteeter: If you want to submit them to me, I will commit them into the SVN repository.

And I have gotten a few emails from people interested in the idea, but I don’t really know how may people might be using it. At this point for me it is all about getting my job done. I have a feeling that the popularity of InstaUp2Date is going to depend on whether the Apple SE’s like Josh Wisenbaker and Mike Bombich take to it and talk to their accounts about it.

[pteeter]

I’ll try to get those catalog files to you today. I’m pre-downloading all of the updates, so the sha-1’s are for PKG installers only. That decrease the usability of what I have?

Well, I’m glad getting your job done has resulted in this tool b/c it really works. If I think of it I’ll harass Josh & Mike (& Joel) about the benefits of InstaUp2Date.

[larkost]

I think to be useful to many people the catalog files are going to have to reference the online versions. The reasons that 10.4 versions are not up is that my organization is switching to 10.5 for all of the new images, so my only 10.4 images are the ones that are on life support until they get re-imaged.

[knowmad]

[QUOTE][u]Quote by: pteeter[/u][p]

larkost – I have to ask…are you (the creator) and me the only people using InstaUp2Date?[/p][/QUOTE]

nope, just the most knowledgeable

[pteeter]

If you need to build multiple images for your company / school / lab /etc. – InstaUp2Date is *the* way to go.

It works. It takes only a little bit of understanding beyond grasping InstaDMG. The developer of it is receptive and helpful…thanks larkost.

[knowmad]

For the record, can you post the link to the SVN repository for instaup2date?

[knowmad]

One more thing. I have been using appfresh to great effect in keeping specific machines up to date and getting current updates for my InstaDMG images. Anyway to incorporate some of that ability into InstaUp2Date?

[larkost]

The InstaUp2Date code is in the same repository (and it SVN, not CVS) as InstaDMG:

http://code.google.com/p/instadmg/

specifically:

http://code.google.com/p/instadmg/source/browse/#svn/trunk/AddOns/InstaUp2Date

Note that the instructions have gotten a little stale, and I am not going to have time to update them for a few weeks at least. I will happily take submissions for updated documentation (or even code).

[larkost]

I have not investigated AppFresh, and am a little uneasy about automating things too much on this part. I certainly want to make updating easier (and a few ideas for a GUI are rumbling about in my head), but part of the whole point on InstaDMG (and by extension InstaUp2Date) is to make images that you can depend on. My idea with catalog files is that we can share them with each other and so everyone knows that they have a good image, and when there is something wrong with one, we can troubleshoot it together. So far there is not enough mass behind this yet, but that will probably have to wait until there is a GUI.

If the system were to automatically pull down updates, then everyone would have their own version of the “base” image, and we are back to square one (albeit a faster square one).

There is probably some middle ground here, but it is going to take some work. I would also suspect that AppFresh would require some convincing to look at the target DMG rather than the host OS. You might want to talk to the developers of AppFresh about that possibility. Once I get through the couple of big projects I have going now I can look back into this direction.

[pteeter]

I’ve played with AppFresh a bit. While I found it’s intent to be admirable – scan Apps and Utils for versions then tell you if you need to update each – it gave me a number of phantom updates on a quite regular basis.

I’m unclear about how AppFresh updates it’s version information.

While it’d be great to completely automate 3rd party application updates, checking VersionTracker daily is the best option in my mind.

[knowmad]

Thank you for the SVN link. As for the appfresh idea, it was more a way of automagically getting the correct download url than anything else.

[Author]

Posts