The maximum team size dictates how many participants can join each team. You selected this when you created your challenge, but you can change it at any time. Here's how:
Open Your Admin Area
Log in to your admin area (your challenge URL with "/admin" at the end).
Open the Current Challenge's "Settings" Section
Click on "Max. Team Size"
Enter the New Maximum Team Size, and Click "OK"
Scroll to the Bottom of the Page and Click "Update"