The underlying reason for the panic about fertility is economics. In our society, there is an assumption that human beings have no value except as a resource to produce wealth. And most of that wealth goes to an undeserving elite that produces almost nothing.
These are sick values. Money is paper and gold is a fairly insignificant metal. And businessmen, politicians and other elites are in a private-public partnership that is destroying the climate for (imagined) personal gain.
There's not a damn thing wrong with a smaller population. Our economy is wrongly designed and should not continue in the same way. We should value human beings and Nature, and not abstract ideas like "the economy" which exists primarily in our heads.
In the same way that immigrants are simply people, the economy is simply the actions of people. Few people isn't a problem. Nor is it a problem that people travel across (stupid, human invented) borders.