Thanks for all your responses. After some further digging I came across the solution to our needs. I created a new role of site administrator for the given site and then for that role under Default Dashboard I added the Users application - as opposed to the Users (global) application which was what was visible before. This only lists users for the current site which is exactly what we need. I do think having two different Users applications is a bit confusing, especially since on the Dashboard the global one isnt named global. If you are looking at the list of users in the interface at the top it says {SiteName} | Users which would lead one to believe that they should be looking at a list of users for that site.
Thanks again for the responses, they did get me thinking in the right direction.