Robots.txt stopped working

Megan Swan asked on January 28, 2020 04:11

Hi there! We have our robots.txt page set up with the custom response web part on a page as described here: https://docs.kentico.com/k11/configuring-kentico/search-engine-optimization/managing-robots-txt

It has worked for years but then stopped working (404 error when visiting robots.txt url) after a server change. We have checked all the settings, including the web.config file. It is all set up correctly. Does anyone know what we may have missed that would stop it working? Thanks :)

robots.txt robots

Recent Answers

Juraj Ondrus answered on January 28, 2020 06:53

What is the new server setup and IIS configuration? Is it the same as the previous one? What is the application pool's managed pipeline mode? Is it set to integrated? What is the target framework set?
Just to be sure, you double-checked that the web.config contains the runAllManagedModulesForAllRequests="true" attribute at the right place?

0 votesVote for this answer Mark as a Correct answer

David te Kloese answered on January 28, 2020 08:53

Are you still connecting to the same Database? Or is that one also migrated?

Is it still the same Domain, or did you also have to change site-settings on de Administration?

0 votesVote for this answer Mark as a Correct answer

Megan Swan answered on January 30, 2020 04:17

Hi guys, thanks for responding. I was mistaken, our server didn't change, we just migrated our exchange server to office365 which has no impact on this issue at all. We are trying to narrow things down, and we were wondering if there should be a physical robots.txt file in the site folder structure, or if kentico does it differently? If so, where should it be located? Thank you :)

0 votesVote for this answer Mark as a Correct answer

David te Kloese answered on January 30, 2020 22:06

You mentioned setting a "custom response web part on a page", so that page should be on the 404 path.

What path has been defined at the Page not found URL under Settings -> Content?

0 votesVote for this answer Mark as a Correct answer