This is probably in here somewhere already, but if it is I can't find it.
I understand there has to be a limit on the number of buildings and Building Permits provides that. I also understand you need to have something you can sell to us (through the Star system) in order to make money. However I don't like the fact you don't get the permits back when you tear down an expired or badly-placed building. Fine, have them to control the maximum, but I think they should restock when you remove buildings. Maybe slowly, say 1 per day or something, but they should come back until they reach your maximum (same as the Settlers do).
For example, I'm nearly stuck already. My forces got kicked (found out the hard way about getting intercepted on the way to bandit camps), I'm out of building permits (built lots of storehouses to improve flow before I found out the permits don't come back) and my fields are about to run out. No wheat = no beer = no troops = no chance (unless I buy some more).