What is robots.txt? Does it have any value for SEO?

Author Topic: What is robots.txt? Does it have any value for SEO?  (Read 8781 times)

Offline richard bransonTopic starter

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 59
  • Karma: 1
    • mcx trading tips
it is used by Google to know what you do NOT want indexed. So if you have content on your website, you do NOT want crawled, indicate that in robots.txt. .



Offline maggyparker07

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 45
  • Karma: 0
Re: What is robots.txt? Does it have any value for SEO?
« Reply #1 on: 05-23-2012, 05:03:53 »
It is having a great valvu if you want to disallow something else it is having no value

Offline Rauny

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 10
  • Posts: 158
  • Karma: 2
    • Cogniter Technologies
Re: What is robots.txt? Does it have any value for SEO?
« Reply #2 on: 05-23-2012, 14:44:20 »
it is used by Google to know what you do NOT want indexed. So if you have content on your website, you do NOT want crawled, indicate that in this file.

Offline ring2012

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 7
  • Posts: 99
  • Karma: 1
    • Quictent
Re: What is robots.txt? Does it have any value for SEO?
« Reply #3 on: 05-24-2012, 01:22:50 »
If you would not like the google robot to crawl the info on your website, then robot.txt can play the important role~~
Welcome to Quictent 8)

Offline john515

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 6
  • Posts: 70
  • Karma: 2
  • Gender: Male
Re: What is robots.txt? Does it have any value for SEO?
« Reply #4 on: 05-24-2012, 04:34:44 »
Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly.  :P


Offline bryanbell

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 7
  • Posts: 107
  • Karma: 2
  • Gender: Male
Re: What is robots.txt? Does it have any value for SEO?
« Reply #5 on: 05-29-2012, 22:59:31 »
Robots.txt is general text file it is use to inform search engine crawler about which page is index and ignore. It is helpful for SEO because it is help to indexing website. This file is located in root directory of website.

Offline fsdqasmia

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 9
  • Posts: 28
  • Karma: 0
    • Buy Backlinks
Re: What is robots.txt? Does it have any value for SEO?
« Reply #6 on: 05-30-2012, 20:27:45 »
This is great for crawling, we should maintain this file  it time to time to increase our ranking
Premium SEO Service & High PR Backlinks

Offline maggyparker07

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 45
  • Karma: 0
Re: What is robots.txt? Does it have any value for SEO?
« Reply #7 on: 05-31-2012, 07:22:41 »
This is great for crawling, we should maintain this file  it time to time to increase our ranking

That file no where relate with the ranking. and no need to manage it time to time

Offline RichardSmith

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 16
  • Posts: 241
  • Karma: 6
Re: What is robots.txt? Does it have any value for SEO?
« Reply #8 on: 06-06-2012, 07:52:55 »
 Hi everyone,

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

Thanks a lot
Richard Smith

 Online Reputation Management


Offline rises5544

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 44
  • Karma: 1
    • SplitGames
Re: What is robots.txt? Does it have any value for SEO?
« Reply #9 on: 06-12-2012, 05:40:03 »
Mostly Robot.txt file use to hide any information from public. But main purpose is not that. Some of the highly renowned websites when want to change in their site without any ugly look while changing its display they create another same site called mirror by using this code.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1514 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1260 Views
Last post 08-23-2016, 06:48:00
by TomClarke
11 Replies
3410 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
1234 Views
Last post 12-28-2016, 05:14:34
by rustyblogger
4 Replies
2106 Views
Last post 12-27-2016, 03:47:18
by damponting44