Crawlers

Author Topic: Crawlers  (Read 2324 times)

Offline WalalayoTopic starter

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 180
  • Karma: 2
    • JavaScript Editor
Crawlers
« on: 11-01-2011, 21:13:17 »
how many levels/layers deep does a search engine crawler go, i have a site map with multiple pages on my site and i think the multiple pages is creating a problem for the crawlers. can anyone offer some insight to this?? thanks in advance


Offline C.Rebecca

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 17
  • Posts: 360
  • Karma: 25
Re: Crawlers
« Reply #1 on: 11-02-2011, 02:41:31 »
A crawler can crawl all the links / pages on your sitemap...

For websites with large number of websites you can create a good internal linking strucutre. A good linking structure means your every webpage is linked to atleast one other web page of your website.
230% more traffic with 12+ Keyword Research Tools

Offline Hogward

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 20
  • Posts: 420
  • Karma: 5
Re: Crawlers
« Reply #2 on: 11-02-2011, 03:04:30 »
Most commonly important product/service page of the site should be within 1 or 2 clicks from the home page. Further deep creates difficulty for both user and crawlers to look for the information.

Offline RaviFitness

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 18
  • Karma: 1
  • Gender: Male
    • outdoor fitness
Re: Crawlers
« Reply #3 on: 11-03-2011, 23:44:33 »
Crawlers in the context of SEO, they are referring to the programs that search engines use to scan and analyze websites in order to determine their importance. Crawlers are very active, and often account for a great deal of the visitors to websites all over the internet.

Offline TM-Ali

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 107
  • Karma: 2
    • SEO Company India
Re: Crawlers
« Reply #4 on: 11-04-2011, 06:16:24 »
web crawler also perform deep link indexing.


Offline Vinil

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 165
  • Posts: 301
  • Karma: 14
  • Gender: Male
  • I can and I Will
    • Boaters Directory
Re: Crawlers
« Reply #5 on: 11-05-2011, 00:01:18 »
Tell me how many pages max.  crawlers can crawl/Also If I have large pages ,then do i need to prepare multiple sitemaps ?

Offline richardmsmith

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 229
  • Karma: 0
  • Gender: Male
    • Trucking App Solution
Re: Crawlers
« Reply #6 on: 09-04-2018, 05:11:02 »
Search engines use internet bots called crawlers to index websites on the World Wide Web and keep their indexes updated.

Offline hiren popat

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 36
  • Karma: 0
    • Yamuna group of companies
Re: Crawlers
« Reply #7 on: 12-07-2018, 03:16:06 »
there are no limits for crawling of the pages, google robots can crawl multiple pages in a time but if you use XML sitemap it helpful for crawling of the pages, sitemap gives a list of pages to the crawler so it could be crawl and index the pages better. the limitation of the sitemap is the 10mb or 50,000 pages.

Offline DeveloperOnRent

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 32
  • Karma: 0
  • Gender: Male
    • DeveloperOnRent
Re: Crawlers
« Reply #8 on: 12-07-2018, 21:59:25 »
Google crawl each and every corner of your website for the web-pages, with the proper sitemap and schema your website will be displayed in the Google search and you can check how many pages indexed in the analytics. 


 

Related Topics

  Subject / Started by Replies Last post
4 Replies
2304 Views
Last post 01-19-2012, 22:54:39
by cpaoutsourcing
6 Replies
2240 Views
Last post 09-26-2018, 07:03:08
by VPSnet
4 Replies
2487 Views
Last post 09-20-2018, 22:24:07
by MVMInfotech18