Crawlers

Started by Walalayo, 11-01-2011, 21:13:17

Previous topic - Next topic

WalalayoTopic starter

how many levels/layers deep does a search engine crawler go, i have a site map with multiple pages on my site and i think the multiple pages is creating a problem for the crawlers. can anyone offer some insight to this?? thanks in advance


C.Rebecca

A crawler can crawl all the links / pages on your sitemap...

For websites with large number of websites you can create a good internal linking strucutre. A good linking structure means your every webpage is linked to atleast one other web page of your website.
230% more traffic with 12+ Keyword Research Tools
  •  


Hogward

Most commonly important product/service page of the site should be within 1 or 2 clicks from the home page. Further deep creates difficulty for both user and crawlers to look for the information.
  •  

RaviFitness

Crawlers in the context of SEO, they are referring to the programs that search engines use to scan and analyze websites in order to determine their importance. Crawlers are very active, and often account for a great deal of the visitors to websites all over the internet.

TM-Ali

web crawler also perform deep link indexing.


Vinil

Tell me how many pages max.  crawlers can crawl/Also If I have large pages ,then do i need to prepare multiple sitemaps ?

richardmsmith

Search engines use internet bots called crawlers to index websites on the World Wide Web and keep their indexes updated.

hiren popat

there are no limits for crawling of the pages, google robots can crawl multiple pages in a time but if you use XML sitemap it helpful for crawling of the pages, sitemap gives a list of pages to the crawler so it could be crawl and index the pages better. the limitation of the sitemap is the 10mb or 50,000 pages.


DeveloperOnRent

Google crawl each and every corner of your website for the web-pages, with the proper sitemap and schema your website will be displayed in the Google search and you can check how many pages indexed in the analytics.