What is the use of Robots.txt?

Author Topic: What is the use of Robots.txt?  (Read 2642 times)

Offline jaysh4922Topic starter

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 10
  • Posts: 341
  • Karma: 0
What is the use of Robots.txt?
« on: 02-01-2016, 23:06:43 »
What is the use of Robots.txt?


Offline ajaymehta588

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 84
  • Karma: 0
  • Gender: Male
    • Rv Industrial Training
Re: What is the use of Robots.txt?
« Reply #1 on: 02-05-2016, 00:26:22 »
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

The structure of a robots.txt is

User-agent:
Disallow:

User-agent are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing.

Offline guptaabhijit318

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 119
  • Karma: 1
  • Gender: Female
  • WIN the SEO Battle!!!
    • Web Development Services
Re: What is the use of Robots.txt?
« Reply #2 on: 02-09-2016, 00:03:57 »
Robots.txt is frequent name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to have the funds for directions about the Web site to Web robots and spiders.

Offline krishnanayak

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 1
  • Posts: 122
  • Karma: 0
  • Gender: Male
    • Fat Cutter Powder
Re: What is the use of Robots.txt?
« Reply #3 on: 02-11-2016, 05:42:50 »
Web site proprietors utilize the/robots.txt record to give directions about their website to web robots; this is known as The Robots Exclusion Protocol. The "Client specialists: *" implies this segment applies to all robots. The "Prohibit:/" tells the robot that it ought not visit any pages on the site.

Offline fix.97

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 64
  • Karma: 0
    • Fikreview
Re: What is the use of Robots.txt?
« Reply #4 on: 02-14-2016, 05:29:57 »
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.


Offline RH-Calvin

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 52
  • Posts: 1018
  • Karma: -1
  • Gender: Male
    • Cheap VPS Hosting
Re: What is the use of Robots.txt?
« Reply #5 on: 02-15-2016, 23:27:57 »
Robots.txt is a text file defined in the website that helps to contain instructions for search engine robots. The file lists webpages which are allowed and disallowed from search engine crawling.

Offline Shikha Singh

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 41
  • Posts: 335
  • Karma: 0
  • Gender: Female
  • solution to start online business
    • solution to start online business
Re: What is the use of Robots.txt?
« Reply #6 on: 02-17-2016, 01:22:29 »
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.yourwebsite.com/robots.txt


Offline designstoredxb

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 27
  • Karma: 0
    • Customized Wallpaper in Dubai
Re: What is the use of Robots.txt?
« Reply #7 on: 02-21-2016, 05:38:33 »
The robots.txt file can be used to block specific pages from the search engine bots so they cannot crawl those pages such admin pages and you can only unblock those pages which you want to show to the visitors publicly.

Offline lilyalvin

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 54
  • Karma: 0
Re: What is the use of Robots.txt?
« Reply #8 on: 02-22-2016, 00:03:53 »
Robots.txt helps prevent the webpages you don't want search engine to crawl or indexed. If you don't want your page or information private than you can use it.


Offline TomClarke

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 17
  • Posts: 896
  • Karma: 0
  • Gender: Male
    • Custom Writing Service & Content Marketplace
Re: What is the use of Robots.txt?
« Reply #9 on: 08-16-2016, 01:44:34 »
A  robots.txt file gives instructions to web robots about the pages the website owner doesn't wish  to be "crawled". For instance, if you didn't want your images to be listed by Google and other search engines, you'd block them using  your robots.txt file. It helps to avoid hаcking problems to a website for  payment gateway sites.
Your ideas, our writings - Writology.com

 

Related Topics

  Subject / Started by Replies Last post
24 Replies
3326 Views
Last post 05-10-2012, 01:38:00
by ring2012
32 Replies
7525 Views
Last post 04-24-2016, 20:52:14
by nguyenmailan
10 Replies
1910 Views
Last post 11-29-2012, 04:09:32
by seocyrusjpr
17 Replies
5490 Views
Last post 04-19-2014, 00:47:04
by ComoliWesta
3 Replies
1725 Views
Last post 09-06-2016, 01:24:52
by TomClarke