A 3-bedroom modular home can be $74k to construct ( http://www.nhmodularhomes.com/prices.htm ). But I've just realized that doesn't include permits, foundation, plumbing, etc, which adds another $75k ( http://www.nhmodularhomes.com/cost_estimates.htm ). So that's $150k. Paying $300-$500k for permits is completely insane. What city is that?

The other bit, of course, is in building materials. I'm guessing the bust has slowed down construction significantly, but you're still likely to have to pay higher costs on things like lumber, brick, sheet rock, et al than you would have to pay in Tennessee or Wyoming.