Go Back   Themers Club : Computers , Mobiles and Web Development Themes > Webmaster forum > Search Engines > Google

Google The search engine. This is for discussion about Google : SEO, PageRank , Serps

Thread Tools Display Modes
Old 05-18-2011, 06:51 PM
sarahbrooks sarahbrooks is offline
Senior Member
Join Date: May 2011
Posts: 184
Rep Power: 13
sarahbrooks is on a distinguished road
Default robots.txt

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site i.e. it is not a firewall, or a kind of password protection and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Reply With Quote
Old 05-29-2012, 05:49 PM
alfie123 alfie123 is offline
Junior Member
Join Date: May 2012
Posts: 5
Rep Power: 0
alfie123 is on a distinguished road

Robot.txt file is very important for the website. It is simple text file and in this file those pages are being included which the web masters don't want to cache by search engines.
Reply With Quote
Old 08-23-2012, 10:54 AM
Paul Jameson Paul Jameson is offline
Junior Member
Join Date: Aug 2012
Posts: 5
Rep Power: 0
Paul Jameson is on a distinguished road

Robots.txt page is important in SEO cause it shows search engine spiders that how many pages has to crawl. Spider crawls only those pages which Robots.txt file navigate it.
Reply With Quote
Old 05-31-2013, 12:25 PM
jenny123 jenny123 is offline
Join Date: Jun 2011
Posts: 51
Rep Power: 13
jenny123 is on a distinguished road

Robot.txt help to owner so that he can hide some official files or admin penal from user. Basically this is the main use of Robot.txt.
Reply With Quote


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

All times are GMT. The time now is 03:21 AM.

Powered by vBulletin® Version 3.7.6
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.