According to the screenshot you provided, your VSTS account is backed by an Azure Active Directory which requires that all users are directory members before they can get access to your Team Services account. So you need to add the user to your AAD first.
"External guest access" is used for external users who are added as guests through Office 365 or added using B2B collaboration by your Azure AD administrator.
Q: Can I control access to my Team Services account for external users in the connected directory?
A: Yes, but only for external users who are added as guests through
Office 365 or added using B2B collaboration by your Azure AD
administrator. These external users are managed outside the connected
directory. To learn more, contact your Azure AD administrator. The
setting below doesn't affect users who are added directly to your
organization's directory.
Refer to this link for more information: Team Services: Access with Azure Active Directory (Azure AD).