How can i allow the spiders to crawl my site ...

Author Topic: How can i allow the spiders to crawl my site ...  (Read 871 times)

Offline kavyasharma561Topic starter

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 118
  • Karma: 0
Hello to folks,

How can i allow the spiders to crawl my site ...


Offline fix.97

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 64
  • Karma: 0
    • Fikreview
Re: How can i allow the spiders to crawl my site ...
« Reply #1 on: 02-29-2016, 03:49:02 »
To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

Offline Shikha Singh

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 41
  • Posts: 335
  • Karma: 0
  • Gender: Female
  • solution to start online business
    • solution to start online business
Re: How can i allow the spiders to crawl my site ...
« Reply #2 on: 03-01-2016, 04:10:46 »
To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

Offline hrishivardhan

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 7
  • Posts: 87
  • Karma: 0
  • Gender: Male
    • Inventory Management Software | Inventory Management System | Order Management Software
Re: How can i allow the spiders to crawl my site ...
« Reply #3 on: 03-01-2016, 04:19:12 »
As you know, Google uses a web crawler named Googlebot to gather information about your website. So it is very important to allow Googlbots to crawl your site. You can do it with txt file name robots.txt. To allow full access of your website your robots.txt file should be like this:
User-agent: *
Disallow:
Orderhive - Multichannel Order & Inventory Management Software

Offline RH-Calvin

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 53
  • Posts: 1093
  • Karma: -1
  • Gender: Male
    • Cheap VPS Hosting
Re: How can i allow the spiders to crawl my site ...
« Reply #4 on: 03-20-2017, 09:30:14 »
You can allow spiders by defining the webpages as allow in robots.txt file. Also for WordPress websites you can go to the settings and check the option 'search engine visibility' to ensure that the spiders can crawl your webpages.


 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1447 Views
Last post 05-31-2010, 03:26:45
by coolman
6 Replies
2647 Views
Last post 12-01-2014, 23:54:34
by AvniShergill
4 Replies
1248 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1042 Views
Last post 08-23-2016, 06:48:00
by TomClarke
3 Replies
1948 Views
Last post 11-29-2016, 07:48:21
by damponting44