Use of Robots.txt

Author Topic: Use of Robots.txt  (Read 6815 times)

Offline rajanippt

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 3
  • Karma: 0
  • Gender: Male
    • corporate picnic
Re: Use of Robots.txt
« Reply #10 on: 07-16-2013, 04:54:24 »
Robots.txt is file is very important to every website. you can allow or disallow the page by the help of robots.txt.
http://www.arounddelhi.net/?page_id=203


corporate picnic


Offline Derickalex

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 25
  • Karma: 0
    • Akshaya Pvt Ltd
Re: Use of Robots.txt
« Reply #11 on: 09-27-2013, 06:40:06 »
Robot.txt is a text file which tells which search engines to crawl the website & which links search engine  should not crawl and index
Affordable flats for sale in Chennai - Akshaya Pvt Ltd

Offline webseomall

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 3
  • Posts: 46
  • Karma: 0
  • Gender: Male
  • Affordable SEO Packages - Best SMO Services
    • Affordable seo services
Re: Use of Robots.txt
« Reply #12 on: 11-18-2013, 03:48:26 »
It is used by webmasters to inform Google that which page need to not index and crawl. by default, all pages are indexed and crawled by search engines. While using robots.txt fine, use it wisely otherwise search engine can harm other important pages by non indexing and non crawling.

use it as -

User-agent: *
Disallow: /
This will instruct all the robots (*) not to index any file from your site (/).

Offline Richard Touchet

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 34
  • Karma: 0
Re: Use of Robots.txt
« Reply #13 on: 11-29-2013, 23:44:11 »
Robot.txt files are the one to by which you can control the contents of your website from getting indexed by the search engine crawlers. You can hide the information from the search engine bot which you don't want that search engine should crawl with the help of Robot.txt.

Offline jesusbond

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 4
  • Posts: 18
  • Karma: 0
    • Feedback Search Engine
Re: Use of Robots.txt
« Reply #14 on: 12-04-2013, 09:23:01 »
Robots.txt is a text file which we can use in ON page optimization to tell the search engines to don't crawl some particular part of site like some folder or url. Generally robots.txt is used for ecommerce site. We can check the robots.txt list from webmaster tool.


Offline seowebagency

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 7
  • Posts: 45
  • Karma: 0
    • e-Celticseo
Re: Use of Robots.txt
« Reply #15 on: 12-05-2013, 00:20:33 »
Think Robots.txt as a traffic signal,which instructs the traffic(robots here)the way to go.It is usually used by webmasters to tell the different search engine crawlers which pages not to crawl.
Thanks 

Offline anilkumartgsb

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 18
  • Karma: 0
Re: Use of Robots.txt
« Reply #16 on: 12-19-2013, 05:08:00 »
Robots.txt is a very important document file for any website. This document contains the information of pages which are allowed to follow by the search engine bots and pages which are disallowed.

For using it you need to keep it in the root directory of your website folder.
My Site: http://www.tgsb.in/seo [nofollow]

Offline ComoliWesta

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 12
  • Karma: 0
    • http://www.webhostingjams.com/reviews/ipage-review/
Re: Use of Robots.txt
« Reply #17 on: 04-19-2014, 00:47:04 »
Robote.txt is useful for hiding your personal content from google crawler. For example, if you want any webpage or content to not show to the world, then mention it to your robot.txt file. Then it will not be crawled by search engine.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
2956 Views
Last post 09-20-2018, 22:24:07
by MVMInfotech18
11 Replies
4419 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2316 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
4 Replies
2002 Views
Last post 12-28-2016, 03:47:42
by wellliving
4 Replies
3066 Views
Last post 12-27-2016, 03:47:18
by damponting44