What is robots.txt? Does it have any value for SEO?

Started by richard branson, 05-19-2012, 02:54:31

Previous topic - Next topic

hieronymusf01

How can robots.txt be used for SEO?

    You can use robots.txt to block Google or Bing from crawling folders or pages you don't want them to crawl.
    Be absolutely sure that you do not block any URL that you want Google to index and display in search results.
    In general, it's better to use a robots meta tag with parameters "noindex, follow" to keep search engines from indexing a page on your site. This way, you won't lose any PageRank or link juice pointed at the page.


nguyenmailan

There are two important considerations when using robots.txt:

- Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.