Allowing online seo analytics tools to crawl my website?

Author Topic: Allowing online seo analytics tools to crawl my website?  (Read 430 times)

Offline janimanTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 7
  • Karma: 0
I have disallowed all the bots to crawl my website because it is under construction.
What I want is just to allow websites like and similar online seo analytics tools to crawl my website.
So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.
The website has some important company data they don't want anyone to access, that's why i can't allow all.

This URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.
« Last Edit: 10-24-2015, 17:18:05 by janiman »


Related Topics

  Subject / Started by Replies Last post
22 Replies
Last post 09-17-2012, 06:29:46
by thelazytrader
2 Replies
Last post 02-20-2014, 00:06:44
by mickyrobert
6 Replies
Last post 12-17-2012, 03:41:18
by bermuda
0 Replies
Last post 11-28-2012, 02:36:04
by abhirampathak3
1 Replies
Last post 01-28-2015, 22:18:14
by Worldeyeglasses