Go Back   Themers Club : Computers , Mobiles and Web Development Themes > Webmaster forum > Search Engines > Google

Google The search engine. This is for discussion about Google : SEO, PageRank , Serps

Reply
 
Thread Tools Display Modes
  #1  
Old 05-18-2011, 06:51 PM
sarahbrooks sarahbrooks is offline
Senior Member
 
Join Date: May 2011
Posts: 184
Rep Power: 7
sarahbrooks is on a distinguished road
Default robots.txt

Robots.txt is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site i.e. it is not a firewall, or a kind of password protection and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Reply With Quote
  #2  
Old 05-29-2012, 05:49 PM
alfie123 alfie123 is offline
Junior Member
 
Join Date: May 2012
Posts: 5
Rep Power: 0
alfie123 is on a distinguished road
Default

Robot.txt file is very important for the website. It is simple text file and in this file those pages are being included which the web masters don't want to cache by search engines.
Reply With Quote
  #3  
Old 08-23-2012, 10:54 AM
Paul Jameson Paul Jameson is offline
Junior Member
 
Join Date: Aug 2012
Posts: 5
Rep Power: 0
Paul Jameson is on a distinguished road
Default

Robots.txt page is important in SEO cause it shows search engine spiders that how many pages has to crawl. Spider crawls only those pages which Robots.txt file navigate it.
Reply With Quote
  #4  
Old 05-31-2013, 12:25 PM
jenny123 jenny123 is offline
Member
 
Join Date: Jun 2011
Posts: 51
Rep Power: 7
jenny123 is on a distinguished road
Default

Robot.txt help to owner so that he can hide some official files or admin penal from user. Basically this is the main use of Robot.txt.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 06:00 PM.


Powered by vBulletin® Version 3.7.6
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.