You population is made up of two things which you can control:
1. Your population limit - the maximum number of settlers you can have
2. You unemployed settlers - how many settlers are available for work
The number of unemployed settlers is displayed in the top bar. The population limit is shown in the population tooltip by hovering over the settler icon. You can also open up a storehouse to see you max population and current population.
To increase population limit you must build or upgrade residences or noble residences or place special buildings such as the white castle which are found on adventures.
The number of unemployed settlers increases over time at a rate of 1 every 15 minutes. You can increase this base rate with a love potion or a cheese sandwich from the merchant/gemshop. The village school independently produces 1 extra settler every 2 hours.
You can also increase the number of unemployed settlers by creating settlers in the provision house using bread. Click provision house -> resources tab -> wayfaring settler / settler family / retired bandits. These caches of settlers are stored in the star menu until you use them on the mayor's house. Stock up before going on a big adventure!