InstaDMG and InstaUp2Date – Questions about multiple HD usage

[dead2sin]

Is there a way to use InstaUp2Date by itself to utilize multiple volumes in order to boost the processing speed?

Or, would I run InstaUp2Date without using the -p command, then just call InstaDMG.bash manually using the -t and -o options?

Thanks,

Nate

[larkost]

The –instadmg-scratch-folder option does exactly that. It does not speed up InstaUp2Date in any way (there is really no place where having multiple disks is going to help with downloading or checksumming), but it does provide this option to InstaDMG for the -p/–process image.

[dead2sin]

Alright, I did a quick test. I ran InstaUp2Date using –instadmg-output-folder as well as –instadmg-scratch-folder. The times listed are only the time it takes for InstaDMG to run, because InstaUp2Date times should be uneffected.

My results are as follows:

Normal InstaDMG run:

1:03:33

Cache Drive InstaDMG Run:

1:03:01

Only 32 seconds faster. Is that normal? None of the drives are Raid 0 (I’ll be setting that up as soon as I get my new HDs for the Mac Pro).

Nate

[larkost]

There is no such thing as normal. It all depends on how fast your second drive is. You don’t talk at all about what your second drive is. If it is a second volume on the same drive, then that is exactly what I would expect (ie: no difference), or if it was a second drive on the same IDE channel. If you tried to use a bus-power drive (ie: a laptop drive), or something over USB then you could actually make things slower.

I will take a look over the code again tonight, to confirm that nothing has messed up that setting, but you have not given enough information to determine if this expected or not.

[dead2sin]

Here is some more info. You are right, I didn’t mention it, but its 4 bare drives in a Mac Pro. 3x 500gb and 1x 80gb. I use one 500gb for the InstaDMG 1.6b2 on 10.6.4, the second 500gb for the tmp cache and the 50gb for the output drive.

Here is something of interest. I ran another run, this time using ONLY –instadmg-scratch-folder pointed to the second 500gb drive and it ran much much faster. I’m going to run another test and see if the 80gb drive is just really slow or what is going on.

My numbers for the –instadmg-scratch-folder run using the 1st 500gb for the InstaDMG folder and running 10.6.4 client and the 2nd 500gb HD being used as the scratch are as follows:

0:29:08 (Two times as fast! Smokin’)

Let me do another run using the 80gb as the scratch drive and we’ll see what the numbers do. If its still slow, then it is clearly my 80gb drive that was the bottleneck. If it isn’t, then perhaps there is something funky going on when I specify both a scratch drive as well as an output drive (although this seems super unlikely). I’ll let you know…I’m 99% sure my 80gb drive is slow.

Nate

[dead2sin]

I can confirm that it was due to a slow drive. My 80gb is much much faster then my 500gb drive, so that is what was causing the slowdown.

500gb as scratch = 1 hour process
80gb as scratch = 30 minute process

HUGE difference. When I go to buy drives for building images next fiscal year I am definitely going to be looking for fast drives. It makes all the difference in the world.

Thanks,

Nate

[larkost]

To amplify this, InstaDMG is almost entirely limited by the speed of the disks. CPU and memory almost don’t come into play at all. An older computer with faster disks can produce an image faster than a newer computer that is more limited by its disks.

I don’t have any solid-state drives to play with at the moment, but I suspect that these would make really disks for image creation.

[dead2sin]

[QUOTE][u]Quote by: larkost[/u][p]To amplify this, InstaDMG is almost entirely limited by the speed of the disks. CPU and memory almost don’t come into play at all. An older computer with faster disks can produce an image faster than a newer computer that is more limited by its disks.

I don’t have any solid-state drives to play with at the moment, but I suspect that these would make really disks for image creation.[/p][/QUOTE]

What is the best setup then? Is there any real advantage to having 3 drives? (One for InstaDMG, one for Scratch and one for Output). I need to consider this before I upgrade the drives on my Mac Pro 🙂

[larkost]

2 drives is the fastest configuration. InstaDMG is always reading from one source, and writing to a second, and at one point reverses the direction. So 2 disks is the optimal solution, you just want to make each one as fast as possible.

[sgstuart]

Hi Nate,
Can you let us know what the drive speeds of each of your HD’s are? Also your 11:42 & 1:07 posts seem to contradict each other on which drive was giving the 30 min speed?

Cutting the time in half is really nice, and I will probably try to do that.

Also, if we do this from a laptop, would Firewire 400 or firewire 800 give any speed increase? I understand already USB2 would not so will not even attempt that.

Thanks,
Steven Stuart

[larkost]

I use a FireWire 800 external 2 disk RAID (so the box is what provides the RAID) in stripe (RAID 0) mode, and see a nice 40% speedup. It cost me less than $200 for 2TB (not that I use any of the space, this is only used for InstaDMG development). I did a bunch of benchmarking and came out with the idea that the FW800 buss was the limiting factor. I am happy with that, as it means I am doing the best I can with the hardware I have. Oh… and the external disk actually benchmarks faster than my internal disk, something I was not expecting. Letting InstaDMG work on both is what gets me the nice speed boost.

[Allister Banks]

As mentioned in the guide, I go from an SSD(housing the entire instadmg dir structure) to a hardware RAID 0 which is striped between two independent eSATA enclosures connected to a HighPoint RocketRaid 2314 card inside a MacPro. I timed it recently with the fastest SSD on the market below $800, a Crucial RealSSD we had in for testing, and it’s faster to output to the RAID than having it do the output to the same drive. I’d like to add a second RAID to go from scratch to output, but need to wait for more enclosures.

Allister

[dead2sin]

[QUOTE][u]Quote by: sgstuart[/u][p]Hi Nate,
Can you let us know what the drive speeds of each of your HD’s are? Also your 11:42 & 1:07 posts seem to contradict each other on which drive was giving the 30 min speed?

Cutting the time in half is really nice, and I will probably try to do that.

Also, if we do this from a laptop, would Firewire 400 or firewire 800 give any speed increase? I understand already USB2 would not so will not even attempt that.

Thanks,
Steven Stuart[/p][/QUOTE]

Good question. I think I got confused 🙂 Let me run the build once more on each drive and I’ll see what we get. I ran it on the 80gb again last night and it took over an hour, so I must of gotten the numbers swapped around.

[dead2sin]

[QUOTE][u]Quote by: Allister[/u][p]As mentioned in the guide, I go from an SSD(housing the entire instadmg dir structure) to a hardware RAID 0 which is striped between two independent eSATA enclosures connected to a HighPoint RocketRaid 2314 card inside a MacPro. I timed it recently with the fastest SSD on the market below $800, a Crucial RealSSD we had in for testing, and it’s faster to output to the RAID than having it do the output to the same drive. I’d like to add a second RAID to go from scratch to output, but need to wait for more enclosures.

Allister [/p][/QUOTE]

Between the Raid 0 array and the SSD, which is faster and by how much? If raid 0 is close to the SSDs, then I’ll just do that because it is slightly cheaper.

Nate

[Allister Banks]

The Crucial nearly saturates the SATA bus on reads, write speed is a fraction of that and definitely slower than two spindles over eSATA. I don’t have hard data at the moment, though, next time I’m in build mode I’ll use the AJA speed measurement tools.

Allister

[Author]

Posts