Robot.txt is good or bad for site?

Author Topic: Robot.txt is good or bad for site?  (Read 10603 times)

Offline john00

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 23
  • Karma: 0
Re: Robot.txt is good or bad for site?
« Reply #10 on: 07-01-2014, 07:31:36 »
Robots.txt file is not necessary for the website. It is used when you do not want to crawl some webpage  from the search engine. If you want to crawl all the webpages to the Google. You have not use the robots.txt file.


Offline danielnash

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 8
  • Posts: 97
  • Karma: 0
  • Gender: Male
    • shipping
Re: Robot.txt is good or bad for site?
« Reply #11 on: 07-02-2014, 06:43:32 »
Robot.txt is good for site because the search engine robots crawl your site, better is your chances of ranking in the SERPs and robot.txt block that pages which is harmful for your site.

Offline jaintech

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 11
  • Posts: 110
  • Karma: 0
  • Gender: Male
    • Website Designing Company
Re: Robot.txt is good or bad for site?
« Reply #12 on: 07-04-2014, 01:40:01 »
Not everyone needs robots.txt. Unless you have some serious content in your website, which you do not want anybody to look into, there is no mandatory need to upload a robots.txt file and not even an empty one. If yours is an e-commerce website and you store your database on it, would you like to disclose the database of your clients’ information to the entire world? In that case robots.txt is necessary.

Offline tailoredsolutions

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 3
  • Posts: 33
  • Karma: 0
  • Gender: Male
  • Leading Joomla & Magento Development Company
    • Tailored Solutions
Re: Robot.txt is good or bad for site?
« Reply #13 on: 07-07-2014, 00:51:40 »
Robots.txt is 100% helpful. It will give instructions to search engine spiders regarding crawling of your website. You must add robots file.

Offline stonewood

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 16
  • Posts: 171
  • Karma: 0
  • Gender: Male
    • Modular Kitchen Manufacturer
Re: Robot.txt is good or bad for site?
« Reply #14 on: 07-31-2014, 03:56:58 »
Robot.txt is good to use because it gives indication to search engine crawler bot that which urls of your website are accessible & which are not. If you want certain urls or directory paths not to be displayed then you can block it by using this txt file. 


Offline SeoDezin

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 11
  • Posts: 214
  • Karma: 1
  • Gender: Male
    • http://cognitive46.wordpress.com/
Re: Robot.txt is good or bad for site?
« Reply #15 on: 07-31-2014, 13:12:29 »
I have never submitted robot.txt for any of my site. I am doing SEO since last 6 years. If a site has good onsite, it is automatically indexed by all major search engines.

Offline SahilFruits

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 109
  • Karma: 0
  • Gender: Male
    • Fresh Fruits Exporters India
Re: Robot.txt is good or bad for site?
« Reply #16 on: 05-14-2015, 04:58:28 »
Robots.txt is only used to block certain directory path/folder. By default everything of the site is allowed, but by using disallow command, we can hide paths from showing in search results.
Interglobe Impex - Fresh Grapes Exporters | Spices

Offline morganlong

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 10
  • Posts: 67
  • Karma: 0
    • XtremeUx pvt ltd
Re: Robot.txt is good or bad for site?
« Reply #17 on: 05-14-2015, 05:27:30 »
Generally Robot.txt file is used which page do you want to index and which page you don't want to index. If there is no Robot.txt file then Google will index all pages of your website.

Offline BrettUK

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 4
  • Posts: 53
  • Karma: 0
  • Gender: Male
  • Put Simply, I ❤ SEO.
Re: Robot.txt is good or bad for site?
« Reply #18 on: 05-15-2015, 04:37:26 »
God, there is no reason you shouldn't have one.


Offline Monikasahu

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 12
  • Posts: 116
  • Karma: 0
Re: Robot.txt is good or bad for site?
« Reply #19 on: 05-19-2015, 02:02:37 »
Robot.txt is very helpful for any online business as it needs to index a website in Google by allowing it to set which web page should be crawled and which not to crawled. By doing this you can assure to get good ranking for your website quickly and even it helps to remove the bad links of your website as well which are of no use.

 

Related Topics

  Subject / Started by Replies Last post
1 Replies
1578 Views
Last post 12-09-2013, 23:57:39
by emedianetwork
1 Replies
1521 Views
Last post 06-08-2015, 09:44:52
by andredevilz
2 Replies
1243 Views
Last post 06-24-2015, 04:40:18
by Herstin
4 Replies
1833 Views
Last post 08-15-2016, 07:23:41
by towertech
18 Replies
5191 Views
Last post 08-22-2016, 04:05:22
by Harikrishna Patel