Use of Robots.txt

Author Topic: Use of Robots.txt  (Read 5857 times)

Offline seocyrusjprTopic starter

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 2
  • Posts: 56
  • Karma: 0
  • Gender: Male
    • Cyrus Technoedge Pvt. Ltd.
Use of Robots.txt
« on: 01-11-2013, 22:37:27 »
Hi friends
I don't know about robots.txt please tell me about it
and how to use it in my website


Offline princysharma1990

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 2
  • Karma: 0
Re: Use of Robots.txt
« Reply #1 on: 01-15-2013, 00:52:44 »
robot.txt file is used by the website owners to instruct the web robots about their site. It allows or disallows the web robots to crawl and index certain files and directories within your site. e.g
User-agent: *
Disallow: /
This will instruct all the robots (*) not to index any file from your site (/).


Offline quain88

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 20
  • Karma: 0
    • SharePirate.Com
Re: Use of Robots.txt
« Reply #2 on: 01-24-2013, 07:04:16 »
if you are using wordpress for CMS then use the plugin google sitemap generator, it will automatically create the necessry robots.txt file

Offline Freddyaiken

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 12
  • Karma: 0
  • Gender: Male
Re: Use of Robots.txt
« Reply #3 on: 01-30-2013, 06:08:24 »
Robot. txt is program which is used by website makers to hide some personal information from search engine. If you apply robot.txt on your site, then google could not able to indexed that things from your website.
Freddy

Offline cortomalteze

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 4
  • Karma: 0
  • Gender: Female
    • Tutor SEO Tools
Re: Use of Robots.txt
« Reply #4 on: 02-06-2013, 04:08:39 »
You can learn more about robots.txt on http://tutor.rs/seo-theory/robots

And then, you can check robots.txt file for a particular website with Tutor Web Review in Seo Basics section.
newbielink:http://tutor.rs [nonactive]


Offline rickywilliams

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 3
  • Karma: 0
    • ictinsite
Re: Use of Robots.txt
« Reply #5 on: 02-13-2013, 01:19:46 »
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
newbielink:http://www.ictinsite.com/internet-marketing [nonactive] | newbielink:http://www.ictinsite.com/web-design [nonactive] | newbielink:http://www.ictinsite.com/seo [nonactive]

Offline jayanta1

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 11
  • Posts: 150
  • Karma: 1
  • Gender: Male
Re: Use of Robots.txt
« Reply #6 on: 02-27-2013, 02:36:33 »
"Robots.txt" is a normal text file. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site. Robots.txt is by no means mandatory for search engines but normally search engines obey what they are asked not to do. The position of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply presume that this site does not have a robots.txt file and therefore they index the whole thing they find along the way.

Offline henry1

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 5
  • Karma: 0
Re: Use of Robots.txt
« Reply #7 on: 03-05-2013, 06:41:29 »
its a file which is use to for indexation of website in search engines.thx
newbielink:https://www.epayments.com/registration?p=964EAEE2CE [nonactive]

Offline seolearnerlive

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 17
  • Karma: 0
  • Gender: Male
    • SEO Company in Mumbai
Re: Use of Robots.txt
« Reply #8 on: 06-11-2013, 00:19:26 »
Hi There,

Robots.txt is a text file which contains rules for the search engine to follow while crawling your website. It tells the search engine which pages not to visit and crawl.
There are two ways of adding robots.txt in your website :

-- Include the robots.txt in your website's root directory
-- You may also add robots meta tag in your index page

Hope this would help,
Live Pages - SEO Services Mumbai


Offline limojohn258

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 23
  • Karma: 0
  • Gender: Male
    • Web Development Company in India
Re: Use of Robots.txt
« Reply #9 on: 07-01-2013, 03:12:12 »
robot.txt file is used by the website owners to instruct the web robots about their site. It allows or disallows the web robots to crawl and index certain files and directories within your site. e.g
User-agent: *
Disallow: /
This will instruct all the robots (*) not to index any file from your site (/).
I agree with you and well said, robot.txt file is not important to use but important when we don't want our site to index in any particular  search engine then it compulsory to ue.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1428 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1198 Views
Last post 08-23-2016, 06:48:00
by TomClarke
11 Replies
3239 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
1135 Views
Last post 12-28-2016, 05:14:34
by rustyblogger
4 Replies
2029 Views
Last post 12-27-2016, 03:47:18
by damponting44