How can i allow the spiders to crawl my site ...

Author Topic: How can i allow the spiders to crawl my site ...  (Read 908 times)

Offline kavyasharma561Topic starter

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 118
  • Karma: 0
Hello to folks,

How can i allow the spiders to crawl my site ...


Offline fix.97

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 64
  • Karma: 0
    • Fikreview
Re: How can i allow the spiders to crawl my site ...
« Reply #1 on: 02-29-2016, 03:49:02 »
To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

Offline Shikha Singh

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 41
  • Posts: 335
  • Karma: 0
  • Gender: Female
  • solution to start online business
    • solution to start online business
Re: How can i allow the spiders to crawl my site ...
« Reply #2 on: 03-01-2016, 04:10:46 »
To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

Offline hrishivardhan

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 7
  • Posts: 87
  • Karma: 0
  • Gender: Male
    • Inventory Management Software | Inventory Management System | Order Management Software
Re: How can i allow the spiders to crawl my site ...
« Reply #3 on: 03-01-2016, 04:19:12 »
As you know, Google uses a web crawler named Googlebot to gather information about your website. So it is very important to allow Googlbots to crawl your site. You can do it with txt file name robots.txt. To allow full access of your website your robots.txt file should be like this:
User-agent: *
Disallow:
Orderhive - Multichannel Order & Inventory Management Software

Online RH-Calvin

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 53
  • Posts: 1099
  • Karma: -1
  • Gender: Male
    • Cheap VPS Hosting
Re: How can i allow the spiders to crawl my site ...
« Reply #4 on: 03-20-2017, 09:30:14 »
You can allow spiders by defining the webpages as allow in robots.txt file. Also for WordPress websites you can go to the settings and check the option 'search engine visibility' to ensure that the spiders can crawl your webpages.


Offline rajeshmehra615

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 64
  • Karma: 0
  • Gender: Male
Re: How can i allow the spiders to crawl my site ...
« Reply #5 on: 03-28-2017, 01:36:33 »
You can allow spiders by defining the webpages as allow in robots.txt file

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1449 Views
Last post 05-31-2010, 03:26:45
by coolman
6 Replies
2654 Views
Last post 12-01-2014, 23:54:34
by AvniShergill
4 Replies
1251 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1046 Views
Last post 08-23-2016, 06:48:00
by TomClarke
3 Replies
1951 Views
Last post 11-29-2016, 07:48:21
by damponting44