Is your website really needs robots.txt?

Author Topic: Is your website really needs robots.txt?  (Read 1018 times)

Offline Die HardTopic starter

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 5
  • Posts: 65
  • Karma: 0
  • Gender: Male
  • No body perfect in this world, I m no body.!
    • Pics Door
Is your website really needs robots.txt?
« on: 09-04-2012, 04:54:10 »
We all know that before Google bot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. But Google webmasters suggest that if you want search engines to index everything in your site. You don't need a robots.txt file. But still lots of website webmaster use robots.txt, while they want search engines to index everything in your site. Google never suggest making a robots.txt file for every website, but webmasters creates.


Offline RajSrivastav

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 34
  • Karma: 2
  • Gender: Male
    • Software Developers UK | Developers Software UK | Web Development Company UK | Web Developers Services UK | Mobile Application Development Company UK | Mobile Apps Developers Uk
Re: Is your website really needs robots.txt?
« Reply #1 on: 09-04-2012, 22:40:54 »
Search engines are indexes websites without adding robots.txt file. When you want single page to be blocked from Search engine crawling adding , no index nofollow on that page would be better

Offline siljajoseph

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 35
  • Karma: 0
    • web design company india
Re: Is your website really needs robots.txt?
« Reply #2 on: 09-05-2012, 03:35:49 »
robots.txt file is not required.That is not to say the robots.txt file isn't useful. You can use it to control which pages the search engines should not crawl, which can be very useful for duplicate content and SEO purposes. There are many useful uses with the robots.txt. But it is not a required element for ranking well.

 

Related Topics

  Subject / Started by Replies Last post
1 Replies
1923 Views
Last post 02-16-2012, 22:40:19
by aprilhomer
24 Replies
4321 Views
Last post 05-10-2012, 01:38:00
by ring2012
31 Replies
9241 Views
Last post 04-24-2016, 20:52:14
by nguyenmailan
10 Replies
2352 Views
Last post 11-29-2012, 04:09:32
by seocyrusjpr
17 Replies
6179 Views
Last post 04-19-2014, 00:47:04
by ComoliWesta