What is robots.txt?

Author Topic: What is robots.txt?  (Read 1962 times)

Offline seoinheritxTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 42
  • Karma: 0
    • iPhone Application Development
What is robots.txt?
« on: 11-22-2012, 03:51:53 »
Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.


Offline abhirampathak3

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 1
  • Posts: 140
  • Karma: 1
  • Gender: Male
  • SEO is Game! PLAY it!! WIN it!!!
    • Web Development Services
Re: What is robots.txt?
« Reply #1 on: 11-22-2012, 04:48:53 »
 Thanks for your post. I think this post will be help full for beginners.
 

Offline judyhenry646

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 31
  • Karma: 1
    • High schools|colleges|University|Education|Projects|Scholarship|forum
Re: What is robots.txt?
« Reply #2 on: 11-22-2012, 23:51:10 »
robots.txt is search engine bots friend. It allows what google search engine can read and what not to read.

Offline celinaaniston86

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 15
  • Karma: 0
  • Gender: Female
  • Center for the New Age Sedona
    • Center for the New Age Sedona
Re: What is robots.txt?
« Reply #3 on: 11-23-2012, 03:51:58 »
Robot.txt is a file which describe the links which web owner does not want to crawl by the search engine. He just put down the root path of the link which he does not want to crawl. Every time search engine robot comes to your website, it check for this file first.
sedonanewagestore.com: New Age CDs [nofollow] __*__ Sedona Vortex Tours [nofollow]

Offline Engagedots CRM

  • Trade Count: (0)
  • Novice
  • *
  • Thank You 0
  • Posts: 0
  • Karma: 0
  • Gender: Female
  • Hello friends
    • Engagedots | CRM software
Re: What is robots.txt?
« Reply #4 on: 11-23-2012, 04:37:17 »
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
newbielink:http://www.engagedots.com/ [nonactive] | newbielink:http://www.engagedots.com/contact-mangement.html [nonactive]


Offline Smith Jones

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 11
  • Posts: 342
  • Karma: 5
  • Gender: Male
Re: What is robots.txt?
« Reply #5 on: 11-23-2012, 06:57:59 »
Hello friends,

Robots.txt file is in the root directory of a website which has been created to direct the activity of search engine crawlers or spiders.

Thanks and regards
Smith Jones

Offline 50stateautoloan

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 78
  • Karma: 0
Re: What is robots.txt?
« Reply #6 on: 11-23-2012, 23:04:26 »
Robots.txt is a text (Not html) file you put on your site to tell search robots which pages you would like them not to visit...:)

Offline titly555

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 1
  • Posts: 111
  • Karma: 0
  • Gender: Female
  • WIN the SEO Battle!!!
    • Quality SEO Services
Re: What is robots.txt?
« Reply #7 on: 11-26-2012, 04:37:27 »
Hi
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages on their website. :)

Offline RickyChhajed

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 51
  • Karma: 0
Re: What is robots.txt?
« Reply #8 on: 11-28-2012, 06:36:27 »
A text file present in the root directory of a website which is used to direct the activity of search engine crawlers. This file is typically used to tell a crawler which portions of the site should be crawled and which should not be crawled.


Offline DavisJoseph

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 98
  • Karma: 0
  • Gender: Male
    • SSL Cerificates Vendor
Re: What is robots.txt?
« Reply #9 on: 11-28-2012, 23:35:23 »
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
Wildcard SSL Certificate will secure unlimited sub domains

 

Related Topics

  Subject / Started by Replies Last post
3 Replies
1821 Views
Last post 09-06-2016, 01:24:52
by TomClarke
4 Replies
1178 Views
Last post 11-23-2015, 03:04:50
by seoroy
4 Replies
973 Views
Last post 08-23-2016, 06:48:00
by TomClarke
5 Replies
617 Views
Last post 12-28-2016, 05:14:34
by rustyblogger
4 Replies
376 Views
Last post 12-27-2016, 03:47:18
by damponting44