Seo Forum

Search Engines => Google Search Optimization => Topic started by: richard branson on 05-19-2012, 02:54:31

Title: What is robots.txt? Does it have any value for SEO?
Post by: richard branson on 05-19-2012, 02:54:31
it is used by Google to know what you do NOT want indexed. So if you have content on your website, you do NOT want crawled, indicate that in robots.txt. .

Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: maggyparker07 on 05-23-2012, 05:03:53
It is having a great valvu if you want to disallow something else it is having no value
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: Rauny on 05-23-2012, 14:44:20
it is used by Google to know what you do NOT want indexed. So if you have content on your website, you do NOT want crawled, indicate that in this file.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: ring2012 on 05-24-2012, 01:22:50
If you would not like the google robot to crawl the info on your website, then robot.txt can play the important role~~
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: john515 on 05-24-2012, 04:34:44
Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly.  :P
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: bryanbell on 05-29-2012, 22:59:31
Robots.txt is general text file it is use to inform search engine crawler about which page is index and ignore. It is helpful for SEO because it is help to indexing website. This file is located in root directory of website.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: fsdqasmia on 05-30-2012, 20:27:45
This is great for crawling, we should maintain this file  it time to time to increase our ranking
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: maggyparker07 on 05-31-2012, 07:22:41
This is great for crawling, we should maintain this file  it time to time to increase our ranking

That file no where relate with the ranking. and no need to manage it time to time
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: RichardSmith on 06-06-2012, 07:52:55
 Hi everyone,

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

Thanks a lot
Richard Smith

 Online Reputation Management (http://www.reputationhead.com/online-reputation-management/)
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: rises5544 on 06-12-2012, 05:40:03
Mostly Robot.txt file use to hide any information from public. But main purpose is not that. Some of the highly renowned websites when want to change in their site without any ugly look while changing its display they create another same site called mirror by using this code.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: moritzbokk on 08-07-2012, 07:45:41
It's especially good, if you have a print preview for each of your sites. So to avoid dublicate contant you can determine with the robots.txt, that all the print previews shouldn't be crawled.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: ljennifer012 on 08-10-2012, 19:33:37
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: milestoneuk on 08-10-2012, 23:12:13
Robot.txt is general text pc computer file it is use to inform seo spider about which web page is index and neglect. It is helpful for SEO because it is help to listing website. This pc computer file is located in root directory of website.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: kradha707 on 08-13-2012, 01:11:53
In order not to crawl our website pages we are giving the information to spiders using robots.txt. If you website is having some sensitive information then you can restrict to crawl those pages with robots.txt.

I do have one more doubt here.

How can i know whether the pages on my site is indexed by google or not, please help me.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: arenakkg on 08-14-2012, 07:11:52
A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.



Thanks...
web design courses (http://www.arenakkg.com/)
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: sureshgupta1 on 08-14-2012, 21:37:20
Robots.txt is a notepad file that use for giving instruction to search engines for which page to crawl or not, Now days it becomes great tool to avoiding duplicates and other things

Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: 12dian04 on 08-15-2012, 00:38:10
robots.txt is a protocol, rather than a command. robots.txt is a search engine visit the Web site you want to view a file. The robots.txt file to tell the spider what files on the server can be viewed.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: salty143 on 08-28-2012, 00:00:42
According to me Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: moritzbokk on 08-29-2012, 04:58:07
with the robots.txt you can handle your linkjuice
for example the contact page is not so important so you tell the crawler with the robots.txt "hey man this page isn't important don't give it some of my linkjuice" ^^
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: aadi14 on 09-16-2012, 23:46:38
Robots.txt Is a file in which we specify/tell Google that which pages he don't want to get indexed.It is used to disallow the pages and by doing it the specified pages are left un-indexed.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: rosicky on 10-06-2012, 13:28:32
Some of the highly renowned websites when want to change in their site without any ugly look while changing its display they create another same site called mirror by using this code.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: johnbob on 02-13-2013, 02:27:36
There are two important considerations when using robots.txt:

- Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: Cathrine on 02-13-2013, 03:01:29
       Robots.txt is a regular text file that through its name has special meaning to the majority of robots on web.By defining a few rules in this  text file,you can instruct robots not to crawl and index certain files,directories within your site,or at all.
        It is very  important file if you want to have a good ranking on search engines.This is useful if you want to hide some personal information or some secret files.Thanks.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: Mike Nielson on 02-13-2013, 23:29:57
robots.txt is a text file webmasters create to instruct robots on how to crawl & index pages on their website. For example your robots.txt file would look something like this:
User-agent:
* Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: davidfleming on 12-03-2014, 22:10:19
Robots.txt is a text (not html) file you place on your site to inform search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but usually search engines obey what they are asked not to do.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: Emmaballet20 on 12-17-2014, 22:34:36
Does Google mind if use robot.txt in excess. I have to use this on every page on my site, could this be harmful for my site.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: guptaabhijit318 on 01-05-2015, 22:46:46
Robots.txt is frequent name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to have the funds for directions about the Web site to Web robots and spiders.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: SanviMalhotra on 08-20-2015, 23:15:03
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: sonth321 on 04-01-2016, 02:51:12
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: TomClarke on 04-12-2016, 03:52:09
robots.txt has two value features. The first and mostly used feature by Google to know what you don't want  indexed. So if you have content on your site that you don't want crawled, you can indicate that in robots.txt. The second feature is that robots.txt is used to point to your XML sitemap. The XML sitemap is the optional but helpful file that helps some search engine understand where content is located on your website.   
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: hieronymusf01 on 04-14-2016, 06:06:44
How can robots.txt be used for SEO?

    You can use robots.txt to block Google or Bing from crawling folders or pages you don't want them to crawl.
    Be absolutely sure that you do not block any URL that you want Google to index and display in search results.
    In general, it's better to use a robots meta tag with parameters "noindex, follow" to keep search engines from indexing a page on your site. This way, you won't lose any PageRank or link juice pointed at the page.
Title: Re: What is robots.txt? Does it have any value for SEO?
Post by: nguyenmailan on 04-24-2016, 20:52:14
There are two important considerations when using robots.txt:

- Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.