should I block robots

Author Topic: should I block robots  (Read 2480 times)

Offline abcdfTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 43
  • Karma: 2
should I block robots
« on: 10-16-2011, 19:29:22 »
Hi

I keep updating my website regularly with some new changes but every time I work on the backend I block robots visits on my website to prevent wrong or incomplete indexing by them in their database.

My question is should I block robots while I am working on the scripts of my website or let them do their work while I am working on source?

Can anyone help me with the correct answer?

Thanks in advace!


Offline leonardw

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 2
  • Karma: 0
Re: should I block robots
« Reply #1 on: 10-16-2011, 23:49:28 »
Yeah! you may able to allow or disallow robots on your site by using robots.txt files. These are very helpful when the site is on maintenance or any changes.







Offline Hogward

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 20
  • Posts: 420
  • Karma: 5
Re: should I block robots
« Reply #2 on: 10-17-2011, 03:41:36 »
while working on certain pages of the site its better to prevent the bots. You can allow only when the pages are ready and need to crawl by the bot.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
3172 Views
Last post 06-01-2010, 02:58:58
by coolrays
1 Replies
2530 Views
Last post 06-04-2015, 11:32:55
by MarcoLeitner
0 Replies
1929 Views
Last post 02-02-2015, 06:03:58
by AvniShergill
0 Replies
1220 Views
Last post 03-02-2015, 00:34:35
by dheerajsaraswat
2 Replies
4074 Views
Last post 11-25-2016, 04:39:23
by CreativeDreamrz