I have set up the robots page and pointed to it in the settings area, yet it does not show up when c

Clyde Timbs asked on October 9, 2017 15:47

I have set up the robots page in Kentico 10 and pointed to the document going in to the settings>URLS and SEO section. However, if I check it using Google's tools or by typing in the robots.txt URL, I am not getting the correct, current version. I am not sure what I need to check at this point to try and remedy this situation. Any pointers?

Thanks, Clyde

robots.txt robots

Recent Answers

Brenden Kehren answered on October 9, 2017 16:01

Do not use the value you entered in settings or even the path tho the page in the content tree, use http://www.domain.com/robots.txt

0 votesVote for this answer Mark as a Correct answer

Clyde Timbs answered on October 9, 2017 16:11

Sorry, that is what I meant. When I go to johnsonu.edu/robots.txt it does not show what is in the actual robots page, instead I get:

User-agent: * Disallow: /

User-agent: Googlebot Noindex: /

In the meantime, I have a user-agent: 8LEGS that is bombarding our site.

0 votesVote for this answer Mark as a Correct answer

Brenden Kehren answered on October 9, 2017 17:04

Is there a physical file in the root of your site? Is the robots.txt page published within Kentico?

0 votesVote for this answer Mark as a Correct answer

Clyde Timbs answered on October 9, 2017 18:48

I set up a robots page in kentico using the pre-defined template with the custom response web part. I then went in to settings and selected that page in the robots.txt path option.

There is a robots.txt file in the root directory that was originally placed there when we were running Kentico 7. It is an exact clone of the information I placed in the custom response web part int he Kentico page I set up referenced above.

Thanks again for your help.

0 votesVote for this answer Mark as a Correct answer

Brenden Kehren answered on October 9, 2017 18:49

If you want to use the dynamic one in the content tree, then you need to remove the one in the root of the site's file system.

1 votesVote for this answer Mark as a Correct answer