I came across this thread while having my own issues with quotas. Sadly I have had no better luck but I did stumble across some interesting observations I thought I would share that might help.
If you go into /var/db/dslocal/nodes/default and find username.plist If you have not set a quota then the field homedirectoryquota does not exist. If you add it as an array and then add a sibling string and for the value you add a value in bytes it shows up in workgroup manager as having a quota. This then persists till restart when all the above go away.
I am not sure if this help but I thought the information was worth passing on.
IFB