Need to clear out old email accounts on 10.4 server

There are a bunch of old accounts still in the mail store directory, but the backups are getting a little unwieldy. I need to pull out those accounts. The accounts were all deleted from Workgroup Manager. Can I just pull out the directories manually, or do I need to use cyradm to remove those to make sure cyrus doesn't get all screwed up?