There isn't a default user interface to only manage people in your group. My recommendation is to either clone or simply modify the existing User Interface to add a WHERE condition (with a macro) that will limit the users returned on the list if the user matches your criteria (just be careful to not limit users who should not be limited).
This can be done through Modules > Membership > User Interfaces > CMS/Administration/Configuration/Users/Users, click on "Customize", go to Properties to see the where condition.
The below will allow all users is the user is at least an administrator, otherwise will limit the Users by some limiting query you will need to define:
MembershipContext.AuthenticatedUser.CheckPrivilegeLevel(UserPrivilegeLevelEnum.Admin) ? "1=1" : "UserID in ([limiting query])"
You may not need to do "code" this may be fully possible through the portal method this way, but here are some links on both methods if you need to create a custom interface.
If you need more info on Permissions on Custom Interfaces, check my blog link below.