I want to VPN to home from work’s open wifi. Here’s the layout:
internet – airport extreme (pppoe outside, nat/DHCP/firewall/DNS inside) – intel mac mini w/os x server 10.6.

The problem is that l2tp won’t connect at all and pptp will connect, but no traffic flows. Yes I read the article about l2tp and BtMM not working together, so I’m trying with PPTP first.

The obvious problems already troubleshot out:
*DNS is all sorted out, no concerns there, as other services on the port forward are fine.
*Port forwards on the AE were configured with “Server Preferences” to enable ssh and VPN forwarding to the mini (192.168.1.2), and “calendarserver” ports are forwarded correctly to another machine. Visually verified using Airport Utility to make sure the IP addresses are set correctly.
*BtMM works. I read about turning it off if you want to use ipsec/l2tp.
*With BtMM off on house macs, l2tp refused to connect at all, even with port forwards set up. Switched to pptp to continue testing.
*PPTP connects in name, but not in practice. The connection appears open but no pinging happens, no traffic moves, etc. Client gets an IP address.
*SSH connects to the mini fine.

Here are the services on the OS X Server and their abbreviated configuration:

DHCP: Off. Letting the Airport Extreme do this.
Firewall: Off. Letting the Airport Extreme do this.
DNS: On. Not sure why, I’m not using it. Letting the Airport Extreme do this.
Open Directory: OD Master. It’s at home, why not. 2 users plus the local machine administrator, 3 total. Me(admin), Me(as user, admin rights), Wife (user user).
iCal: On, not exposed through port forwarding yet.
Remote Management: on, mainly so I can BtMM to my main desktop and make adjustments while troubleshooting.
VPN service (server):
l2tp on, addresses 192.168.1.200-220 (not inside the DHCP range from the AE), kerberos auth, shared secret set
pptp on, addresses 192.168.1.190-199 (not inside the DHCP range from the AE), directory service ms-chapv2.
client info: DNS server 192.168.1.1 == AE, search domain == my domain, network routing == 192.168.1.0 mask 255.255.255.0 private

VPN client (at work):
server address set,
account == Me (as user, admin rights),
encryption 128 only,
auth == password (set),
send all traffic over VPN (which is fine for what I want to do),
no vpn on demand,
tcp == ppp ipv4, automatically ipv6,
DNS empty,
no proxies.