Never underestimate a text file!

Ok, I may be laughed out of the forum for this one, but I thought I might share a story of my weekend with you. Late Friday night, I recieved word that email accounts weren't working on our local server. I went in via webmail and recieved some messages about not being able to create a folder. After logging in to the server I noticed that the mail store directories didn't seem to be in very good shape. I figured I'd end up having to use the Admin tools to take care of the problem, so unfortunately I had to wait till I got in this morning (Car's in the shop, and a 20 minute drive turns into a much longer walk). Anyway, after spending some time looking around and trying to figure out what's going on, one of the employees here mentioned that they weren't able to drop files on the server either. I thought it was a bit odd, "df -h" showed that indeed, the storage drive was completely full. So, now the task came down to finding out where all this room had gone to.

Now, I should mention at this point that I had set up a backup system last thursday that would send a report of its run to me by saving the output of Rsync to a logfile and writing that into the body of an email. When I checked the output Friday morning I noticed that there were errors regarding permissions. I figured out that the script needed to be set to run as root rather than admin (something I should have thought of right away, but alas, hindsight is 20/20 while foresight is not always as clear).

Back to this morning, while searching directories I noticed that the home share was fairly large and upon closer inspection the admin directory was the one taking up all of the room. So, a "du -h" on that directory showed that the "logs" directory inside was taking up all of the room. By this time I was beginning to realize what happened, but when I actually came accross the 40G log file (largest text file I've ever seen in my life) sitting in that directory I was still quiet dumbfounded.

So the moral of the story, I suppose, is that computers will never cease to surprise the people that work with them and to keep that in mind when troubleshooting .