Interesting. Because I used Blank Site template which had nothing in it. I went into the Settings and the Web Farm option was set to Disabled.
Message: License for feature 'Webfarm' not found.
Exception type: CMS.DataEngine.LicenseException
...
The web app can scale out, but it's set to a single instance for now (can scale out to 10 according to the plan). This error seems to be something that has to do with the Azure Marketplace Kentico implementation. I guess I could stand up an instance locally and then deploy it to Azure.
This issue will cause real errors to be hard to locate.