Looking for suggestions on large file hierarchy

[kennyj]

Hello,

We have a group that utilizes a series of external applications to manage incoming files and process them. The file/folder hierarchy is pretty complex and causes some major issues when attempting to back it up, and when attempting an fsck on the raid volume that houses it. Because of directory structures like this, it causes backup apps such as rsync to take forever to crawl the trees… also it takes hours and hours to perform an fsck if something were to go wrong.

Currently, this folder structure is contained on the same volume as other data. I have a project to replace the whole system and storage and will most likely end up splitting this off into it’s own volume to try and alleviate it causing problems for other production data.

Does anyone out there have suggestions as to how to maintain complicated structures like this in OS X? I think this may be pushing it for HFS+… really wish we had zfs support in 10.6 right now.

The folder structure is layed out similar to this… the following is an example of one path, there are hundreds of directories with a similar layout
** artwork/ – 84 folders
*** artwork/600000/ – 358 folders
**** artwork/600000/600513/ – 4 folders
The structure for artwork/6000000/600513/ looks like this:
|—–/1579812C
|——-Fonts
|——-Links
|——-job_info
|—–/1579812H
|—–/Customer_E-Files
|——-EHBS-56609
|——-EHBS-56610
|——-EHBS-56611
|——-EHBS-56612
|——-EHBS-78937
|—–/HBuilderImages
|——-EFS_56610
|——-EFS_56611
|——-EFS_56612
|——-EFS_78937
|——-REPEATCOVER_1404580
|——-REPEATCOVER_1503968
|——-REPEATCOVER_1579812
|——-REPEAT_1503968
|——-REPEAT_1579812

As always, thanks for any help/suggestions you can provide,
Ken

[Author]

Posts