should I block robots

Author Topic: should I block robots  (Read 1716 times)

Offline abcdfTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 43
  • Karma: 2
should I block robots
« on: 10-16-2011, 19:29:22 »
Hi

I keep updating my website regularly with some new changes but every time I work on the backend I block robots visits on my website to prevent wrong or incomplete indexing by them in their database.

My question is should I block robots while I am working on the scripts of my website or let them do their work while I am working on source?

Can anyone help me with the correct answer?

Thanks in advace!


Offline leonardw

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 2
  • Karma: 0
Re: should I block robots
« Reply #1 on: 10-16-2011, 23:49:28 »
Yeah! you may able to allow or disallow robots on your site by using robots.txt files. These are very helpful when the site is on maintenance or any changes.







Offline Hogward

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 20
  • Posts: 420
  • Karma: 5
Re: should I block robots
« Reply #2 on: 10-17-2011, 03:41:36 »
while working on certain pages of the site its better to prevent the bots. You can allow only when the pages are ready and need to crawl by the bot.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
2367 Views
Last post 06-01-2010, 02:58:58
by coolrays
1 Replies
2093 Views
Last post 06-04-2015, 11:32:55
by MarcoLeitner
0 Replies
1484 Views
Last post 02-02-2015, 06:03:58
by AvniShergill
0 Replies
972 Views
Last post 03-02-2015, 00:34:35
by dheerajsaraswat
2 Replies
3427 Views
Last post 11-25-2016, 04:39:23
by CreativeDreamrz