Site structure
Version 3.x > Site structure > Robots.txt View modes: 
User avatar
Member
Member
m00nstar - 9/24/2008 7:44:35 PM
   
Robots.txt
Hello!

Just a quick one. How/where would I create a robots.txt file in Kentico? I can easily add a file to the server itself if not supported within the CMS, but where?

Cheers
Daleth.

User avatar
Kentico Support
Kentico Support
kentico_jurajo - 9/25/2008 3:25:25 AM
   
RE:Robots.txt
Hello,

You need to place the file to the root of the web project. Moreover, this file have to be created manually - there is no option how to create it with Kentico CMS right now.

Best Regards,
Juraj Ondrus

User avatar
Member
Member
m00nstar-hotmail - 9/25/2008 3:57:46 PM
   
RE:Robots.txt
Thank you.

I'm wondering where the project root is. This piece of documentation says it should be here: \Inetpub\wwwroot\<web project>

But I can't find the folder I need there. In a standard install, is there anyother place it could be? Does it matter what version you are using?

Thanks again!

User avatar
Kentico Support
Kentico Support
kentico_jurajo - 9/26/2008 3:02:42 AM
   
RE:Robots.txt
Hi,

Which folder you are not able to find? The example is just the default setting. If you have installed your web into the root there is no <web project>. The meaning of the brackets is that it is optional and the folder name could be custom.

You only need to place the file to the root of your web project.

Best Regards,
Juraj Ondrus

User avatar
Member
Member
m00nstar-hotmail - 9/29/2008 4:27:47 PM
   
RE:Robots.txt
Thanks!

User avatar
Member
Member
mcutter-global-id-group - 12/16/2008 11:58:30 AM
   
RE:Robots.txt
When I upload a file robots.txt to the root level then it gets served by the url ~/robots.aspx, and not robots.txt. I can easily put in a url rewrite from robots.txt to robots.url but am wondering if there is a way to do this within Kentico itself.

User avatar
Kentico Support
Kentico Support
kentico_radekm - 12/22/2008 1:45:45 AM
   
RE:Robots.txt
Hello,

there is no native way how to do it within Kentico itself in present time. We plan to add this feature into some of future releases of Kentico CMS.

Best Regards,
Radek Macalik

User avatar
Member
Member
Vispi - 4/24/2010 5:25:21 PM
   
RE:Robots.txt
Hi guys

Just wanted clarification on the robots.txt

Would a robots.txt file with the following content
User-agent: *
Allow: /default.aspx
Disallow: /


Will this file serve the website and all the pages created within the CMS, while blocking the remaining folders such as CMSPages, etc?

Also, my blog entries on the website do not appear to be getting indexed by search engines. How can I enable them to be generated on the Sitemap or allow them to be crawled by search engines?

Many thanks,
Vispi

User avatar
Kentico Support
Kentico Support
kentico_radekm - 5/25/2010 7:02:56 AM
   
RE:Robots.txt
Hello.

It should be like this for your purpose:

User-agent: *
Disallow: /CMSPages/

As for how to show cms.blog document in Site map web part, you only need to set web part´s property "Document types" like this: CMS.MenuItem;CMS.Blog;

Best Regards,
Radek Macalik

User avatar
Certified Developer 8
Certified Developer 8
manoj-pulldigital - 1/18/2011 11:29:11 AM
   
RE:Robots.txt
Hi,

Another really esay solution is to add a robot.txt as a file type in the cms desk. To do this you need to allow file type under the root document type in site manager> documenttypes search for 'root'

Upload your robot.txt file

You than need to allow the url extention property of .txt in the url property of the uploaded robot.txt file.

This is the only solution that I could think of if you have multiple sites on a single instance of kentico... interested in hearing any other thoughts...


User avatar
Certified Developer v7
Certified  Developer v7
jason-datausa - 8/1/2011 4:16:12 PM
   
RE:Robots.txt
Hello,

You can also use the rewrite module if you're hosting your website on IIS 7 to create a different robots.txt file for each website. It's very easy to set up! If you're unfamiliar with the rewrite module, I've written a more detailed post on how to accomplish this here: http://www.kenticosolutions.com/Developer-Tips/Tip/July-2011/Multiple-Websites-and-Robots-txt-no-problem.aspx.

User avatar
Kentico Support
Kentico Support
kentico_zdenekc - 8/4/2011 8:51:38 AM
   
RE:Robots.txt
Hi Jason,

That's nice, thank you for sharing your solution with us here.

www.kenticosolutions.com is definitely great information source for all Kentico CMS developers and users.

Regards,
Zdenek Cetkovsky