When you upgrade to level 5 it takes a full week. It'll take a full 4 weeks to regain your loss of production, since the buildings being upgraded don't produce. That may not be a problem with farms and cutters, since you usually have lot's of those, but for smelters and smiths it just isn't worth it. Not to mention the resources it takes to upgrade. If the upgrade would give +2 yield for level 4 and +3 yield for level 5, it would be better.
And while I'm on the subject upgrading mines is pretty useless too. My level 1 mines unbuffed delivers more than my upgraded smelters (notice the plural) can handle. If you don't have an enourmos amount of tools, that you can't get rid of by trading, there's no point in upgrading. If the mines would yield a little more ore instead of or in addition to increased production it would be more usefull. I'm a gemmer, so i could easily restock my mines with golden parrots and copper diviners, but I don't see the point when the other end of the production line is full. I'll get more worth for my money by having lot's of geo's.