Please see 7:7. How can Moshe say the jews were the fewest? There were 601730 men between 20 and 60. That’s quite a large population. For the jews to be smaller than all the other nations there would need to have been over 21 million people in Israel and over 9 million people among Moav, Ammon and Edom.
I can’t tell you how many people were living in the land at the time, but the conditions then were differant than they are now, and the land was much more fertile. However Rashi and Rabeinu Bachya address your question. Rashi says that what it says that you are not the biggest but the smallest nation, means that you are are a nation that is not haughty, and that you lessen yourself instead of making yourselves as if you are the biggest and greatest. Rabeinu Bachya explains a little differantly. Even if you wouldn’t be the biggest nation but only the smallest nation, still H-shem choose you.