While Juraj is correct, I can say we have set up multiple sites as you're explaining without issue. We have not experienced this: "Nesting one of the applications under the other causes configuration errors and prevents certain parts of the system from working correctly."
What it seems you have an issue with is your virtual directory on your main website. In IIS you'd set your site up like so:
So your domain would be domain.com/cms/admin to log in. Using simple IIS setup, you'll need that sub-application to get the CMS to run properly under the main MVC site. The only other way around this would be to set this up the same way and add a rewrite in the MVC site