What is Robots.txt?

Author Topic: What is Robots.txt?  (Read 5214 times)

Offline merryscanlan

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 1
  • Posts: 9
  • Karma: 1
  • Gender: Female
    • eBazR
Re: What is Robots.txt?
« Reply #20 on: 04-27-2012, 05:07:02 »
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
newbielink:http://www.ebazr.com [nonactive] | newbielink:http://www.ebazr.com/Auction-Registration.aspx [nonactive] | newbielink:http://classified.ebazr.com [nonactive]


Offline kelly

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 10
  • Posts: 115
  • Karma: 1
Re: What is Robots.txt?
« Reply #21 on: 04-27-2012, 23:03:19 »
Robot.txt is the text file which tells crawler that which part of the website has been index or not.
Syntax:-
User-agent: *
Disallow: /temp/

Offline pinnaclewebsolutions

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 33
  • Karma: 0
Re: What is Robots.txt?
« Reply #22 on: 04-30-2012, 00:50:06 »
I think Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

Offline seolinkcreation

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 6
  • Posts: 173
  • Karma: 2
Re: What is Robots.txt?
« Reply #23 on: 05-09-2012, 11:25:18 »
Basically a short twenty-two hide someone's personal information is a code. Or is it a secret information.due Robot.txt page or search engine files to hide files, file "robots.txt the" can not access the page using the publicly.companies will not open.

Offline ring2012

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 7
  • Posts: 99
  • Karma: 1
    • Quictent
Re: What is Robots.txt?
« Reply #24 on: 05-10-2012, 01:38:00 »
It is an agreement, if you do not want to google spider to crawl your link, you can use robot.txt. briefly
Welcome to Quictent 8)


 

Related Topics

  Subject / Started by Replies Last post
4 Replies
3275 Views
Last post 09-20-2018, 22:24:07
by MVMInfotech18
11 Replies
4512 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2384 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
4 Replies
2081 Views
Last post 12-28-2016, 03:47:42
by wellliving
4 Replies
3206 Views
Last post 12-27-2016, 03:47:18
by damponting44