There are a couple of ways to get there:
- go to particular module settings under the Modules app, open sites and remove sites, those do not need that module
- go to particular module settings under the Modules app, open User interface and set visibility condition to false under general tab
- Configure permissions/UI personalization, so only particular roles can see/access particular module
I'd go with the last option as it is most flexible and improves UX for different roles. First two options are acceptable in case you don't need that module at all, however system might use some module on the background, so you don't even know it is being used, so be careful with it.