Google Crawl rate

Author Topic: Google Crawl rate  (Read 3894 times)

Offline tmcconnellTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 1
  • Posts: 6
  • Karma: 0
Google Crawl rate
« on: 05-15-2012, 16:51:48 »
What does everyone with dynamically updated websites set their crawl rate to? I don't want to waste too much bandwidth on googlebot but want recent caches in google.
newbielink:http://allweblinks.zxq.net [nonactive] - Free and reciprocal web directory with 24 hour approval.


Offline moritzbokk

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 25
  • Karma: 0
Re: Google Crawl rate
« Reply #1 on: 08-07-2012, 08:34:18 »
it's a myth that you can determine when google's bots should come to crawl your site. It doesn't work
I mean think about it, people would say the bots they should come every 30 minutes

Offline cisritu

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 44
  • Karma: 0
  • Gender: Female
    • Virtual Assistant Service
Re: Google Crawl rate
« Reply #2 on: 08-21-2012, 06:49:23 »
What does everyone with dynamically updated websites set their crawl rate to? I don't want to waste too much bandwidth on googlebot but want recent caches in google.

I think we can not escape to an easy alternative or any things such as called  tricks.  People do use trick but using tricks is a risky task!
If you want your site to crawled faster you have compromise with the bandwidth.
Get the Virtual Assistant Service and Trained virtual assistant  for your business![/co]

Offline Allenbrayan

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 3
  • Posts: 37
  • Karma: 0
  • Gender: Male
Re: Google Crawl rate
« Reply #3 on: 08-28-2012, 04:43:15 »
You need to update your content often and regularly (and ping Google once you do). Try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).

Offline moritzbokk

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 25
  • Karma: 0
Re: Google Crawl rate
« Reply #4 on: 08-29-2012, 05:11:48 »
Once Google noticed that you update the content of your site regularly the crawler will come more


Offline seoguru7

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 2
  • Posts: 61
  • Karma: 3
  • Gender: Female
Re: Google Crawl rate
« Reply #5 on: 08-29-2012, 07:47:46 »
Google is crawling  your site on the basis of traffic, pagerank, Updates, Your website's quality and Content etc.... There are other factors also but it is not revealed yet. I am also looking forward to it.

Offline ZabBZaa

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 11
  • Karma: 0
  • SEO Life
    • SEO
Re: Google Crawl rate
« Reply #6 on: 09-09-2012, 10:47:54 »
Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

  • Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
  • Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
  • Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
  • Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
  • Get more back links from regularly crawled sites.
  • Adjust the crawl speed via Google Webmaster tools.
  • Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
  • Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
  • Make sure you have unique title and meta tags for each of your pages.
  • Monitor Google crawl rate for your site and see what works and what not.
O0

Offline brandsmith

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 2
  • Posts: 52
  • Karma: 0
  • Gender: Male
    • Eilersmarketing
Re: Google Crawl rate
« Reply #7 on: 09-11-2012, 04:17:27 »
If you are looking for fast crawling rate then you must update fresh content that entice to every visitors read up to down and also give provide best information related their search query.

Offline GMT

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 4
  • Karma: 0
Re: Google Crawl rate
« Reply #8 on: 09-11-2012, 14:14:20 »
I know that googlebot tends to ignore the refresh time in sitemaps so yeah as others said you can't really determine how often googlebot will crawl your content. Though if your site is popular or you ping google upon updates(say for a blog) it can increase the frequency. I had a blog that Googlebot visited constantly daily.


Offline apsarabanu

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 28
  • Karma: 0
Re: Google Crawl rate
« Reply #9 on: 09-14-2012, 05:40:22 »
Hi everyone,
           If you have a high crawl rate, it means you have lots of fresh content that will be added to Google’s index fast, i think if you have a way to make minor changes to all your pages frequently, you will see the crawl rate increase.

 

Related Topics

  Subject / Started by Replies Last post
20 Replies
7191 Views
Last post 06-23-2015, 10:53:36
by Talukdar
2 Replies
1594 Views
Last post 02-24-2015, 01:19:21
by nancyisabell
0 Replies
621 Views
Last post 07-20-2015, 04:44:46
by Zinavocouk
2 Replies
545 Views
Last post 02-16-2017, 00:30:47
by amitkedia
1 Replies
757 Views
Last post 02-19-2017, 21:19:20
by wellliving