Robot.txt is good or bad for site?

Author Topic: Robot.txt is good or bad for site?  (Read 10460 times)

Offline Shikha Singh

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 52
  • Posts: 309
  • Karma: 0
  • Gender: Female
  • solution to start online business
    • solution to start online business
Re: Robot.txt is good or bad for site?
« Reply #20 on: 03-05-2016, 00:08:39 »
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.By having the robots file, even if it's empty, will keep the error log from getting clogged with those errors thereby making it easier to find broken links and missing files when checking the log. It will also save some bandwith because whenever the server encounters a "page not found" error it consumes bandwith.


 

Related Topics

  Subject / Started by Replies Last post
1 Replies
1492 Views
Last post 12-09-2013, 23:57:39
by emedianetwork
1 Replies
1459 Views
Last post 06-08-2015, 09:44:52
by andredevilz
2 Replies
1178 Views
Last post 06-24-2015, 04:40:18
by Herstin
4 Replies
1727 Views
Last post 08-15-2016, 07:23:41
by towertech
18 Replies
5053 Views
Last post 08-22-2016, 04:05:22
by Harikrishna Patel