What is the use of Robots.txt?

Author Topic: What is the use of Robots.txt?  (Read 4191 times)

Offline sanjaysing373

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 8
  • Karma: 0
Re: What is the use of Robots.txt?
« Reply #10 on: 08-26-2016, 04:52:28 »
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
Technomoz is the Best content writing service In India.
« Last Edit: 08-26-2016, 05:00:48 by sanjaysing373 »


Offline mukeshkr

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 6
  • Posts: 43
  • Karma: 0
Re: What is the use of Robots.txt?
« Reply #11 on: 10-05-2016, 06:39:48 »
robot.txt is exclusion protocol or simply robot.txt is a standard file used by websites to communicate with web crawlers and other web robots. The file indicate those url to search engine who did't want to show in google search result page.

 

Related Topics

  Subject / Started by Replies Last post
24 Replies
4797 Views
Last post 05-10-2012, 01:38:00
by ring2012
31 Replies
10287 Views
Last post 04-24-2016, 20:52:14
by nguyenmailan
10 Replies
2641 Views
Last post 11-29-2012, 04:09:32
by seocyrusjpr
17 Replies
6609 Views
Last post 04-19-2014, 00:47:04
by ComoliWesta
5 Replies
2194 Views
Last post 04-12-2016, 23:39:58
by jaysh4922