Population Growth

[PlutonianEmpire]

If any of you have visited the soft sci-fi universe I'm creating (link is in the signature), you'll notice that even though there's FTL (faster than light) travel, and that the ally has a lot of people too, there's a lot of time covered in between colonization times of the bodies in the Solar system. I do it 'cuz I assume it takes time to build up a human population after being reduced to a few million. My question is, how do I make sure the population growth is realistic?