How can i allow the spiders to crawl my site ...

[kavyasharma561]

Hello to folks,

How can i allow the spiders to crawl my site ...

[fix.97]

To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

[Shikha Singh]

To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

[hrishivardhan]

As you know, Google uses a web crawler named Googlebot to gather information about your website. So it is very important to allow Googlbots to crawl your site. You can do it with txt file name robots.txt. To allow full access of your website your robots.txt file should be like this:
User-agent: *
Disallow:

[RH-Calvin]

You can allow spiders by defining the webpages as allow in robots.txt file. Also for WordPress websites you can go to the settings and check the option 'search engine visibility' to ensure that the spiders can crawl your webpages.

[apwebsolutions]

in a standard wordpress site, as long as you are not blocking robots, they should eventually be able to automatically crawl your site. A great way to make it a quicker process is by sharing your pages on social media websites (to create links which the bots will follow). You can also try submitting your sitemap or URLs for Google bot to crawl.

[JohnVilson]

Hi there,

You have to put allow in the robots.txt and spider will crawl it and index it.

For an instance: 

User:*
Allow:/

[sarahmaxwell789]

use

User-agent: *
Disallow:

in your robots.txt file to allow spiders to crawl your site.