Seo Forum

Search Engine Optimization => SEO Basics => Topic started by: Die Hard on 09-04-2012, 04:54:10

Title: Is your website really needs robots.txt?
Post by: Die Hard on 09-04-2012, 04:54:10
We all know that before Google bot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. But Google webmasters suggest that if you want search engines to index everything in your site. You don't need a robots.txt file. But still lots of website webmaster use robots.txt, while they want search engines to index everything in your site. Google never suggest making a robots.txt file for every website, but webmasters creates.
Title: Re: Is your website really needs robots.txt?
Post by: RajSrivastav on 09-04-2012, 22:40:54
Search engines are indexes websites without adding robots.txt file. When you want single page to be blocked from Search engine crawling adding , no index nofollow on that page would be better
Title: Re: Is your website really needs robots.txt?
Post by: siljajoseph on 09-05-2012, 03:35:49
robots.txt file is not required.That is not to say the robots.txt file isn't useful. You can use it to control which pages the search engines should not crawl, which can be very useful for duplicate content and SEO purposes. There are many useful uses with the robots.txt. But it is not a required element for ranking well.