The problem was that I did not do the following step. The site was going step-by-step all the way from version 8.2 to 11. If I understand correctly a new version 10 install doesn't need this. Hopefully this helps someone else.
Below was taken from the Kentico 11 upgrade doc:
Required .NET Compiler Platform
Kentico 11 requires the .NET Compiler Platform ("Roslyn") compiler to run. Installations of Kentico 10 or newer contain the Microsoft.CodeDom.Providers.DotNetCompilerPlatform NuGet package by default, which provides the required functionality.
However, if your project is upgraded from Kentico 9 or older, you may be missing the package. In these cases, you need to install the package before applying the upgrade:
Open your Kentico solution in Visual Studio.
Right-click the main web project in the Solution Explorer and select Manage NuGet Packages.
Install the Microsoft.CodeDom.Providers.DotNetCompilerPlatform NuGet package (version 1.0.2 or newer).