should I block robots

Author Topic: should I block robots  (Read 1485 times)

Offline abcdfTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 43
  • Karma: 2
should I block robots
« on: 10-16-2011, 19:29:22 »
Hi

I keep updating my website regularly with some new changes but every time I work on the backend I block robots visits on my website to prevent wrong or incomplete indexing by them in their database.

My question is should I block robots while I am working on the scripts of my website or let them do their work while I am working on source?

Can anyone help me with the correct answer?

Thanks in advace!


Offline leonardw

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 2
  • Karma: 0
Re: should I block robots
« Reply #1 on: 10-16-2011, 23:49:28 »
Yeah! you may able to allow or disallow robots on your site by using robots.txt files. These are very helpful when the site is on maintenance or any changes.







Offline Hogward

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 20
  • Posts: 420
  • Karma: 5
Re: should I block robots
« Reply #2 on: 10-17-2011, 03:41:36 »
while working on certain pages of the site its better to prevent the bots. You can allow only when the pages are ready and need to crawl by the bot.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
2083 Views
Last post 06-01-2010, 02:58:58
by coolrays
1 Replies
1931 Views
Last post 06-04-2015, 11:32:55
by MarcoLeitner
0 Replies
1247 Views
Last post 02-02-2015, 06:03:58
by AvniShergill
0 Replies
782 Views
Last post 03-02-2015, 00:34:35
by dheerajsaraswat
2 Replies
3094 Views
Last post 11-25-2016, 04:39:23
by CreativeDreamrz