The Amish community didn't really have problems with other communities, though there were attacks on them by Indians (Hostetler Massacre), but that wasn't unique to them.
The reason for the low estimate of Amish in the early 20th century is that the Amish community was always small. They also struggled to maintain their way of life during the Industrial Revolution. For example, between the 1860s-1880s almost 2/3rds split from the Amish and integrated into U.S culture. Some groups liberalized in the early-20th century and mid-20th century and slowly integrated into U.S culture. The Amish retention rate was also much lower in the past then it is today. Today, the retention rate is close to 85-90%.
The first wave of Amish emigrants left for the U.S in the mid-1700s. The number who emigrated in the 1700s was around 500. In the early 1800s, a few hundred emigrated to Canada. That was followed by a larger wave to the mid-western U.S states that numbered several thousands.