Allowing online seo analytics tools to crawl my website?

Author Topic: Allowing online seo analytics tools to crawl my website?  (Read 527 times)

Offline janimanTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 7
  • Karma: 0
I have disallowed all the bots to crawl my website because it is under construction.
What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.
So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.
The website has some important company data they don't want anyone to access, that's why i can't allow all.

Error
This URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.
« Last Edit: 10-24-2015, 17:18:05 by janiman »


 

Related Topics

  Subject / Started by Replies Last post
22 Replies
3822 Views
Last post 09-17-2012, 06:29:46
by thelazytrader
2 Replies
2084 Views
Last post 02-20-2014, 00:06:44
by mickyrobert
6 Replies
2290 Views
Last post 12-17-2012, 03:41:18
by bermuda
0 Replies
777 Views
Last post 11-28-2012, 02:36:04
by abhirampathak3
1 Replies
887 Views
Last post 01-28-2015, 22:18:14
by Worldeyeglasses