What is robots.txt? Does it have any value for SEO?

Author Topic: What is robots.txt? Does it have any value for SEO?  (Read 10367 times)

Offline rosicky

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 12
  • Karma: 0
Re: What is robots.txt? Does it have any value for SEO?
« Reply #20 on: 10-06-2012, 13:28:32 »
Some of the highly renowned websites when want to change in their site without any ugly look while changing its display they create another same site called mirror by using this code.
party time..!


Offline johnbob

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 23
  • Karma: 0
Re: What is robots.txt? Does it have any value for SEO?
« Reply #21 on: 02-13-2013, 02:27:36 »
There are two important considerations when using robots.txt:

- Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

Offline Cathrine

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 3
  • Posts: 86
  • Karma: 0
Re: What is robots.txt? Does it have any value for SEO?
« Reply #22 on: 02-13-2013, 03:01:29 »
       Robots.txt is a regular text file that through its name has special meaning to the majority of robots on web.By defining a few rules in this  text file,you can instruct robots not to crawl and index certain files,directories within your site,or at all.
        It is very  important file if you want to have a good ranking on search engines.This is useful if you want to hide some personal information or some secret files.Thanks.

Offline Mike Nielson

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 3
  • Posts: 18
  • Karma: 0
    • Seoorangecountyca
Re: What is robots.txt? Does it have any value for SEO?
« Reply #23 on: 02-13-2013, 23:29:57 »
robots.txt is a text file webmasters create to instruct robots on how to crawl & index pages on their website. For example your robots.txt file would look something like this:
User-agent:
* Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/


Offline davidfleming

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 13
  • Posts: 211
  • Karma: 0
    • XGC Media
Re: What is robots.txt? Does it have any value for SEO?
« Reply #24 on: 12-03-2014, 22:10:19 »
Robots.txt is a text (not html) file you place on your site to inform search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but usually search engines obey what they are asked not to do.
#1 PROVEN Provider of Bulk Accounts & Services: Gmails~Yahoos~Youtube~Instagram~Pinterest~Twitter! Go to http://www.XGCmedia.com


Offline Emmaballet20

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 10
  • Posts: 172
  • Karma: 0
    • Guest Posting India
Re: What is robots.txt? Does it have any value for SEO?
« Reply #25 on: 12-17-2014, 22:34:36 »
Does Google mind if use robot.txt in excess. I have to use this on every page on my site, could this be harmful for my site.

Offline guptaabhijit318

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 11
  • Posts: 114
  • Karma: 1
  • Gender: Female
  • WIN the SEO Battle!!!
    • Web Development Services
Re: What is robots.txt? Does it have any value for SEO?
« Reply #26 on: 01-05-2015, 22:46:46 »
Robots.txt is frequent name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to have the funds for directions about the Web site to Web robots and spiders.

Offline SanviMalhotra

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 6
  • Posts: 57
  • Karma: 0
  • Gender: Female
    • Best E-commerce Platform in India
Re: What is robots.txt? Does it have any value for SEO?
« Reply #27 on: 08-20-2015, 23:15:03 »
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

Offline sonth321

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 4
  • Posts: 141
  • Karma: 0
    • Viral Loop Review
Re: What is robots.txt? Does it have any value for SEO?
« Reply #28 on: 04-01-2016, 02:51:12 »
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.


Offline TomClarke

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 53
  • Posts: 844
  • Karma: 0
  • Gender: Male
    • Custom Writing Service & Content Marketplace
Re: What is robots.txt? Does it have any value for SEO?
« Reply #29 on: 04-12-2016, 03:52:09 »
robots.txt has two value features. The first and mostly used feature by Google to know what you don't want  indexed. So if you have content on your site that you don't want crawled, you can indicate that in robots.txt. The second feature is that robots.txt is used to point to your XML sitemap. The XML sitemap is the optional but helpful file that helps some search engine understand where content is located on your website.   
Your ideas, our writings - Writology.com

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
2715 Views
Last post 09-20-2018, 22:24:07
by MVMInfotech18
11 Replies
4261 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2227 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
4 Replies
1901 Views
Last post 12-28-2016, 03:47:42
by wellliving
4 Replies
2848 Views
Last post 12-27-2016, 03:47:18
by damponting44