Kinda off topic, and probably controversial, but the reason most people get irked about population is that a lot of the vocal organizations pushing it are doing so for *financial gain while too many of the really loud individuals are closeted
xenophobes/racists. For the most part, environmental impacts are caused by
the wealthy, to the point where the average American emits as much in the way of GHGs as 1000+ Kenyans. Roughly speaking,
the ~1.4 billion people living in the EU, Japan, and the U.S. cause as much damage to the environment as the other ~5.3 billion people in the world. If we ignore national boundaries and instead focus on wealth we would probably see that the richest hundreds of millions on the planet cause the same impact as the the other 6+ billion people do. Of course, the point isn't that we would all be better w/o wealthy people, but that our problems are due to specific facets of our economy as opposed to our population as a whole. Looking at the rich illustrates this because they consume far more than most of the world.
*Fossil fuels alone account for tens of trillions of dollars in global wealth every year, not to mention the economic winners and losers from the restriction of other environmentally damaging activities. A lot of the supposed controversy about population growth IMO is just astroturfing/PR campaigns by wealth interests who would be hurt by moving to more sustainable ways of doing things.