Robot.txt is good or bad for site?

Author Topic: Robot.txt is good or bad for site?  (Read 9785 times)

Offline NahinTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 25
  • Karma: 0
  • Gender: Female
    • http://clippingpath.in/digital-makeup/
Robot.txt is good or bad for site?
« on: 02-11-2014, 00:03:37 »
Today I know about the robot.txt and it also knows that we can set it for allow or disallow. But I have no idea that to using this is good or bad for site. Because if we don't use than Google never index the site so way to find out in search again if we add it than it will be hamper when Google will update any algorithm like panda, hummingbird, penguin.  :(


Offline emedianetwork

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 10
  • Posts: 175
  • Karma: 1
  • Gender: Male
  • Professional search engine optimization company
    • Easy Media Network
Re: Robot.txt is good or bad for site?
« Reply #1 on: 02-11-2014, 02:01:34 »
Using Robot.txt file is not mandatory and could be used based on your desire. It actually stops or encourages links to be crawled by Google. By default, Google indexes and follows all links in a website. If you are using it, there is no problem at all, but use it wisely, otherwise, it can stop indexing some important pages by mistake.

Offline kumar29

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 12
  • Posts: 152
  • Karma: 1
  • Gender: Male
    • Hotels
Re: Robot.txt is good or bad for site?
« Reply #2 on: 03-02-2014, 11:19:50 »
Robots.txt is a text file placed on the root of the server to tell the search engine crawlers that there are restrictions for the crawlers in crawling the targeted site, We can perform various tasks with robots.txt, some of them are prevent search engine crawlers from crawling the site, stop crawlers from specific pages/directories etc,

Offline abhi_garg

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 8
  • Posts: 138
  • Karma: 0
Re: Robot.txt is good or bad for site?
« Reply #3 on: 03-04-2014, 21:50:33 »
Robot.txt is good & bad that all depends condition ! Basically Robot.txt is useful because it helps to Google for understanding index links.

Offline jaintech

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 10
  • Posts: 111
  • Karma: 0
  • Gender: Male
    • Website Designing Company
Re: Robot.txt is good or bad for site?
« Reply #4 on: 03-05-2014, 23:38:25 »
Good bots, like Google’s spider, crawl your site to index it for search engines or provide some other symbiotic use. Others spider your site for more nefarious reasons such as stripping out your content for republishing, downloading whole archives of your site or extracting your images. Hence Robot.txt is good for site.


Offline jayanta1

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 12
  • Posts: 150
  • Karma: 1
  • Gender: Male
Re: Robot.txt is good or bad for site?
« Reply #5 on: 04-08-2014, 01:20:17 »
The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file. Websites with sub-domains generally need a Robots.txt file for each sub-domain, all so that information that is not viewable to the public is not picked for a keyword search.

Offline jobtardis

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 10
  • Karma: 0
    • jobtardis
Re: Robot.txt is good or bad for site?
« Reply #6 on: 04-28-2014, 06:02:29 »
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to advising cooperating web crawlers and other web robots about accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

Offline arindamdutta16

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 22
  • Posts: 422
  • Karma: 0
  • Gender: Male
  • WIN the SEO Battle!!!
    • Web Design Services
Re: Robot.txt is good or bad for site?
« Reply #7 on: 04-30-2014, 00:52:24 »
 The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.
 

Offline jaysh4922

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 30
  • Posts: 360
  • Karma: 0
Re: Robot.txt is good or bad for site?
« Reply #8 on: 06-23-2014, 23:00:46 »
Its very important for a website Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content. !-!


Offline arunavaseo

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 3
  • Posts: 21
  • Karma: 0
Re: Robot.txt is good or bad for site?
« Reply #9 on: 06-30-2014, 22:57:24 »
Robot.txt is very useful in SEO. If you want to block any pages of your website which you think it could be harmful for your site, you can use robot.txt to avoid indexing by Google crawler.

 

Related Topics

  Subject / Started by Replies Last post
1 Replies
1229 Views
Last post 12-09-2013, 23:57:39
by emedianetwork
1 Replies
1258 Views
Last post 06-08-2015, 09:44:52
by andredevilz
2 Replies
942 Views
Last post 06-24-2015, 04:40:18
by Herstin
4 Replies
1347 Views
Last post 08-15-2016, 07:23:41
by towertech
18 Replies
4375 Views
Last post 08-22-2016, 04:05:22
by Harikrishna Patel