Only a team 'Admin' (the person who created the team) can cancel pending invites.

On your challenge website, login and visit the "My Team" page.

On the pending invite row, click on the 3 dots (more) icon, then choose "Cancel Invite"

You will be asked to confirm if you want to cancel the invite - click "OK" to cancel.