The front end, live site is no longer served by Kentico but by your MVC5 or .NET Core app. This means this is fully under your control and you can use common techniques in MVC to build the robots.txt page. This is what I found on Google:
And there are many more, there is no single correct approach. From Kentico point of view, if you want to get the pages dynamically, I would use the page's API to retrieve the pages from Kentico and then apply your logic to build the robots.txt output.