Is it possible to setup the Kentico 13 living site as a web application under a site

It is possible to set your site up this way, however, I believe Kentico recommends against doing so. Your .net core site would be set up as the primary site with a presentation URL of https://domain.com. Then you'd add your Kentico instance as a virtual application under the main site running asp.net 4.8 framework and the cms admin domain would be domain.com/<your virtual application name>

You can run the apps as virtual IIS apps, just the admin app and live site app need to run side-by-side. It is not recommended to nest the apps. So, If you main web site in the IIS points to a blank folder, then it is OK. I have my IIS setup like:

_website (pointing to blank physical folder)
__live site virtual app (pointing to the front end app folder)
__admin virtual app (pointing to the CMS folder)

So, this means I have domain.com/liveSite and domain.com/adminapp URLs.
The important thing is not to nest the projects/apps - IIS can detect nested web.config files and then some iheritance is applied so some web.config settings can clash, are (not) applied differently, etc.

In the case of using .net core, nested web.config files is not an issue though. Even with MVC5 app you can make modifications to the CMS web.config to allow both to work just as you would have to with any other nested asp.net app.

@Brenden - I would still not do it even in MVC5 as some Kentico functionality may not work - like page builder, form builder. Making it work takes much more effort than just running the apps side-by-side. Moreover, I would say that there are very little use cases, where you have your production web running in a virtual app so it is better to have the dev/text/staging setup as much similar as possible to the production IIS setup and then just change license keys/URL setup.