How can i allow the spiders to crawl my site ...

Author Topic: How can i allow the spiders to crawl my site ...  (Read 2038 times)

Offline kavyasharma561Topic starter

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 11
  • Posts: 123
  • Karma: 0
Hello to folks,

How can i allow the spiders to crawl my site ...


Offline fix.97

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 4
  • Posts: 63
  • Karma: 0
    • Fikreview
Re: How can i allow the spiders to crawl my site ...
« Reply #1 on: 02-29-2016, 03:49:02 »
To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

Offline Shikha Singh

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 52
  • Posts: 323
  • Karma: 0
  • Gender: Female
  • solution to start online business
    • solution to start online business
Re: How can i allow the spiders to crawl my site ...
« Reply #2 on: 03-01-2016, 04:10:46 »
To allow crawling of your WP site you need to go to "admin panel >> Settings >> Reading" right before the "Save Changes" button is a checkbox with the label Search Engine Visibility make sure this is unchecked.

Offline hrishivardhan

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 15
  • Posts: 92
  • Karma: 0
  • Gender: Male
    • Inventory Management Software | Inventory Management System | Order Management Software
Re: How can i allow the spiders to crawl my site ...
« Reply #3 on: 03-01-2016, 04:19:12 »
As you know, Google uses a web crawler named Googlebot to gather information about your website. So it is very important to allow Googlbots to crawl your site. You can do it with txt file name robots.txt. To allow full access of your website your robots.txt file should be like this:
User-agent: *
Disallow:
Orderhive - Multichannel Order & Inventory Management Software

Offline RH-Calvin

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 86
  • Posts: 1240
  • Karma: 0
  • Gender: Male
    • Cheap VPS Hosting
Re: How can i allow the spiders to crawl my site ...
« Reply #4 on: 03-20-2017, 09:30:14 »
You can allow spiders by defining the webpages as allow in robots.txt file. Also for WordPress websites you can go to the settings and check the option 'search engine visibility' to ensure that the spiders can crawl your webpages.


Offline apwebsolutions

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 19
  • Posts: 77
  • Karma: 1
  • Gender: Male
    • A.P. Web Solutions
Re: How can i allow the spiders to crawl my site ...
« Reply #5 on: 04-05-2017, 07:37:28 »
in a standard wordpress site, as long as you are not blocking robots, they should eventually be able to automatically crawl your site. A great way to make it a quicker process is by sharing your pages on social media websites (to create links which the bots will follow). You can also try submitting your sitemap or URLs for Google bot to crawl.
Converting websites into revenue-generating machines !-! http://www.apwebsolutions.com.au

Offline JohnVilson

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 13
  • Posts: 128
  • Karma: 0
    • Website Designing Company in Mumbai
Re: How can i allow the spiders to crawl my site ...
« Reply #6 on: 04-05-2017, 23:36:31 »
Hi there,

You have to put allow in the robots.txt and spider will crawl it and index it.

For an instance: 

User:*
Allow:/

Offline sarahmaxwell789

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 3
  • Posts: 22
  • Karma: 0
    • Online Marketing Company
Re: How can i allow the spiders to crawl my site ...
« Reply #7 on: 04-06-2017, 02:59:51 »
use

User-agent: *
Disallow:

in your robots.txt file to allow spiders to crawl your site.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1686 Views
Last post 05-31-2010, 03:26:45
by coolman
6 Replies
3291 Views
Last post 12-01-2014, 23:54:34
by AvniShergill
4 Replies
1633 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1329 Views
Last post 08-23-2016, 06:48:00
by TomClarke
3 Replies
3247 Views
Last post 11-29-2016, 07:48:21
by damponting44