What is Robots.txt?

Started by sreelavanya, 03-26-2012, 00:46:35

Previous topic - Next topic

merryscanlan

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
newbielink:http://www.ebazr.com [nonactive] | newbielink:http://www.ebazr.com/Auction-Registration.aspx [nonactive] | newbielink:http://classified.ebazr.com [nonactive]
  •  


kelly

Robot.txt is the text file which tells crawler that which part of the website has been index or not.
Syntax:-
User-agent: *
Disallow: /temp/


pinnaclewebsolutions

I think Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note "Please, do not enter" on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
  •  

seolinkcreation

Basically a short twenty-two hide someone's personal information is a code. Or is it a secret information.due Robot.txt page or search engine files to hide files, file "robots.txt the" can not access the page using the publicly.companies will not open.

ring2012

It is an agreement, if you do not want to google spider to crawl your link, you can use robot.txt. briefly