InstaUp2Date/InstaDMG Cache after catalogs?

[dead2sin]

I was just curious if it would be possible/feasible to have a feature in InstaDMG/Up2Date that would allow images to be caches after processing a catalog. The idea is, the Vanilla image might not change much for a month or three, so why not cache the image after vanilla has been processed and then pick that cached image up on the next catalog file in the build chain. This would allow the time between builds to be a lot faster (with testing mainly in mind). Is this something that is on the roadmap and is it even possible?

Thanks!

Nate

[Allister Banks]

Hey Nate,

There is a remnant of the idea to compress the cached base image that is still in the instadmg.bash script, but was commented out at some point. It would certainly be helpful with just the 10.5.8 combo update, since that’s a good amount of time to sink each session, and it is conceivable that the same process can be followed at any point in the build train, great idea. Please file the feature-request issue at googlecode, although we have our work cut out for us when it comes to other issues in 10.5 IMHO. Thanks,

Allister

[dead2sin]

I would only care if this worked on 10.6 🙂 I don’t really need the functionality in 10.5.

Nate

[larkost]

There are now two questions here:

1) Why not cache the vanilla image (base image + combo + other updates)?

The answer here is that I do have a design in mind to do essentially this, but it requires that at run-time the system knows that nothing in the process up to the the point of caching has changed. Unfortunately with pure InstaDMG this idea was too complex, and I had to drop it. Making sure that the image is correct trumps speed (although the latter is also a goal). But with the merge of InstaUp2Date that I have had in the works for a long time now this becomes possible because everything is checksummed (so I can validate things), and InstaUp2Date/InstaDMG can tell explicitly when things change. I am still working out a lot of the details, but rest assured that this is planned.

For the moment though, the best thing to do is to grab a second fast external disk and use the option to use two drives durring image creation. This really drives down the time.

2) Allister saw the compression of the base image that is still hanging around in code. I tried to compress the base image a long time ago, figuring that I could make it take up a bit less room, and probably gain a bit of speed out of it (since disk I/O is what kills us). But something in it blew up every time, and at the time I was under some pressure to get the other features (base image caching) running, so it got put off. I should go back and validate that again….

[dead2sin]

[QUOTE][u]Quote by: larkost[/u][p]There are now two questions here:

1) Why not cache the vanilla image (base image + combo + other updates)?

The answer here is that I do have a design in mind to do essentially this, but it requires that at run-time the system knows that nothing in the process up to the the point of caching has changed. Unfortunately with pure InstaDMG this idea was too complex, and I had to drop it. Making sure that the image is correct trumps speed (although the latter is also a goal). But with the merge of InstaUp2Date that I have had in the works for a long time now this becomes possible because everything is checksummed (so I can validate things), and InstaUp2Date/InstaDMG can tell explicitly when things change. I am still working out a lot of the details, but rest assured that this is planned.

For the moment though, the best thing to do is to grab a second fast external disk and use the option to use two drives durring image creation. This really drives down the time.

2) Allister saw the compression of the base image that is still hanging around in code. I tried to compress the base image a long time ago, figuring that I could make it take up a bit less room, and probably gain a bit of speed out of it (since disk I/O is what kills us). But something in it blew up every time, and at the time I was under some pressure to get the other features (base image caching) running, so it got put off. I should go back and validate that again….[/p][/QUOTE]

A Thought I had had in regards to item #1 was using checksums. Would this scenario be possible:

Store a checksum of the post-catalog cached image within the catalog for it (I.E., Vanilla catalog has a checksum in it for the cached image that has vanilla already processed). So when it processes, InstaUp2Date would look at the checksum and compare it to the cached Vanilla catalog image to make sure its still the same. Then, if you want to reprocess vanilla, when you update it, just delete the chucksum and that was instaup2date reprocesses it. That way, it ensures accuracy while adding speed and still allows you to reprocess on command.

Would that even be possible or desired?

Nate

[larkost]

You can’t store the checksum of the final vanilla image, since there are a lot of things in the image that get the date that the image is created (the receipts for instance). So two vanilla images that are functionally equivalent are not going to have the same checksum either internally, or of the DMG (side note: I do want to create a tool that I can use to create checksums for images where two runs would result in the same checksum, and use this in validation. But that is necessarily going to be slow, and would really only serve to help me tell when I have broken something during development).

I do plan on using a checksum to do this, but one that combines the checksums of the inputs (installer dvd + installers + installer choices files, including settings and possibly an InstaDMG “compatibility number”, done in order). I have sketched this out already, but it absolutely requires something like InstaUp2Date to drive it, while at the same time requiring that the same component be driving the selection of the cached image. I am tentatively calling this idea “break-points”, and am still figuring out how they are going to work (beyond automatically setting one after the combo update).

[dead2sin]

[QUOTE][u]Quote by: larkost[/u][p]You can’t store the checksum of the final vanilla image, since there are a lot of things in the image that get the date that the image is created (the receipts for instance). So two vanilla images that are functionally equivalent are not going to have the same checksum either internally, or of the DMG (side note: I do want to create a tool that I can use to create checksums for images where two runs would result in the same checksum, and use this in validation. But that is necessarily going to be slow, and would really only serve to help me tell when I have broken something during development).

I do plan on using a checksum to do this, but one that combines the checksums of the inputs (installer dvd + installers + installer choices files, including settings and possibly an InstaDMG “compatibility number”, done in order). I have sketched this out already, but it absolutely requires something like InstaUp2Date to drive it, while at the same time requiring that the same component be driving the selection of the cached image. I am tentatively calling this idea “break-points”, and am still figuring out how they are going to work (beyond automatically setting one after the combo update).[/p][/QUOTE]

Sounds good to me! I can’t wait to see what you come up with in the future. Its not a huge deal as you said due to the fact that is is already pretty speedy, but it would be a nice feature for later down the road.

Nate

[Author]

Posts