What is Robots.txt?

Started by sreelavanya, 03-26-2012, 00:46:35

Previous topic - Next topic

onlinemotorcycle

You can't use robot.txt file on free blogging site like blogger or wordpress.some site put that code in your sub pages for doesn't want to show pages to anybody.


pinnaclewebsolutions

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note "Please, do not enter" on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
  •  


preeti22

Robots.txt file is nothing but a text file which allows robots what to scan specific pages or the whole website.
  •  

fizzer

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.
Check out my new hosting forum ,Money Maker Forum and Teen Forum.
  •  

lora93

This is a notepad file which helps to remove our web pages from Google index or if we want that google bot could not  crawl our  web pages from Google spider then we place these links in robots.txt file.


developers

Information of that pages or files, folders, images we want not to ne crawled or indexed by search engines are placed in robot.txt.Robot.txt file is for search engines which helps search engines in knowing which pages not to be followed by them.
elgg developers [nofollow] | elgg open source [nofollow] | elgg plugins [nofollow]
  •  

serviceuncle

There is a hidden, relentless force that permeates the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here.
newbielink:http://www.serviceuncle.com.au [nonactive]
!-!Submit your business website with first online newbielink:http://b2b%20marketplace [nonactive] for service industry. Paid and free business listing options are available with Service Uncle to im
  •  

salty143

Robots.txt is a file which tells search engine's crawler that which pages he can crawl and he page are not to crawl.


simpsonsbill

some companies use this Meta tag in your some pages for doesn't crawl by google bot.
  •  


TM-Ali

Robots.txt is used to disallow the search engine's robots for indexing.