Use of Robots.txt

Started by seocyrusjpr, 01-11-2013, 22:37:27

Previous topic - Next topic

seocyrusjprTopic starter

Hi friends
I don't know about robots.txt please tell me about it
and how to use it in my website


princysharma1990

robot.txt file is used by the website owners to instruct the web robots about their site. It allows or disallows the web robots to crawl and index certain files and directories within your site. e.g
User-agent: *
Disallow: /
This will instruct all the robots (*) not to index any file from your site (/).

  •  


quain88

#2
The robots.txt file is a text file placed in the root directory of your website that gives instructions to web robots (also known as web crawlers) about which pages or files they should or should not access. This file allows you to control how search engines index and crawl your site.

To create and use a robots.txt file for your website, follow these steps:

1. Create a new text file using a plain text editor like Notepad or TextEdit.
2. Include specific directives to allow or disallow certain bots from accessing certain parts of your website. For example:
   - To allow all bots to access all content, use:
     User-agent: *
     Disallow:
   - To disallow all bots from accessing all content, use:
     User-agent: *
     Disallow: /
3. Save this file as "robots.txt".
4. Upload the robots.txt file to the root directory of your website using an FTP client or through your website's hosting control panel.

It's important to note that while robots.txt can give instructions to well-behaved bots, it doesn't serve as a method to secure your website or keep sensitive information private. It's mainly a way to communicate with web crawlers. Also, keep in mind that not all web crawlers respect the directives in the robots.txt file, so it's not a foolproof method for controlling access to your website.

Freddyaiken

Robot. txt is program which is used by website makers to hide some personal information from search engine. If you apply robot.txt on your site, then google could not able to indexed that things from your website.
Freddy
  •  

cortomalteze

You can learn more about robots.txt on http://tutor.rs/seo-theory/robots

And then, you can check robots.txt file for a particular website with Tutor Web Review in Seo Basics section.
newbielink:http://tutor.rs [nonactive]
  •  


rickywilliams

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
newbielink:http://www.ictinsite.com/internet-marketing [nonactive] | newbielink:http://www.ictinsite.com/web-design [nonactive] | newbielink:http://www.ictinsite.com/seo [nonactive]
  •  

jayanta1

"Robots.txt" is a normal text file. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site. Robots.txt is by no means mandatory for search engines but normally search engines obey what they are asked not to do. The position of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply presume that this site does not have a robots.txt file and therefore they index the whole thing they find along the way.

henry1

its a file which is use to for indexation of website in search engines.thx
newbielink:https://www.epayments.com/registration?p=964EAEE2CE [nonactive]
  •  


seolearnerlive

Hi There,

Robots.txt is a text file which contains rules for the search engine to follow while crawling your website. It tells the search engine which pages not to visit and crawl.
There are two ways of adding robots.txt in your website :

-- Include the robots.txt in your website's root directory
-- You may also add robots meta tag in your index page

Hope this would help,
Live Pages - SEO Services Mumbai
SEO Company in Mumbai [nofollow] | SEO Services Mumbai [nofollow]
  •  


limojohn258

Quote from: princysharma1990 on 01-15-2013, 00:52:44
robot.txt file is used by the website owners to instruct the web robots about their site. It allows or disallows the web robots to crawl and index certain files and directories within your site. e.g
User-agent: *
Disallow: /
This will instruct all the robots (*) not to index any file from your site (/).
I agree with you and well said, robot.txt file is not important to use but important when we don't want our site to index in any particular  search engine then it compulsory to ue.
  •