Robot.txt is good or bad for site?

Author Topic: Robot.txt is good or bad for site?  (Read 10604 times)

Offline Shikha Singh

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 52
  • Posts: 307
  • Karma: 0
  • Gender: Female
  • solution to start online business
    • solution to start online business
Re: Robot.txt is good or bad for site?
« Reply #20 on: 03-05-2016, 00:08:39 »
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.By having the robots file, even if it's empty, will keep the error log from getting clogged with those errors thereby making it easier to find broken links and missing files when checking the log. It will also save some bandwith because whenever the server encounters a "page not found" error it consumes bandwith.


 

Related Topics

  Subject / Started by Replies Last post
1 Replies
1578 Views
Last post 12-09-2013, 23:57:39
by emedianetwork
1 Replies
1521 Views
Last post 06-08-2015, 09:44:52
by andredevilz
2 Replies
1243 Views
Last post 06-24-2015, 04:40:18
by Herstin
4 Replies
1833 Views
Last post 08-15-2016, 07:23:41
by towertech
18 Replies
5192 Views
Last post 08-22-2016, 04:05:22
by Harikrishna Patel