It's great to have the option to spend gems on hurrying things up in-game, but would anyone really consider spending 21,834 gems to upgrade their bookbinder from level 4 to level 5? This is higher than the cost of 365 premium days!
There are many other examples around the game, everyone has their favourite ludicrously-priced item.
If gem spend on trivial items e.g. speeding up the item being produced in the barracks or provision house were cut to micro-level payments (e.g. 5 gems to complete 25 cannoneers), I would suggest that the total gem spend (excluding inadvertent clicks, but that's another subject) would actually rise... 500 x 5 gems is more than 0 x 200 gems.
I'm not suggesting the cost of buildings etc. should be changed, just trivial items.