Google spider not good enough

Author Topic: Google spider not good enough  (Read 1725 times)

Offline manvbfTopic starter

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 1
  • Posts: 123
  • Karma: 1
Google spider not good enough
« on: 11-01-2011, 03:59:46 »
Well, I feel like I've been forced to take matters into my own hands, and build my own site search. I've tried quite a few different things, from site maps to mod rewrite. It seems that google decides how many pages it wants to index on a site based on its own algorithm that determines the importance of a site. That's fine for Google, but the users of my site suffer. There are too many "supplemental" results and unindexed pages.

Take phpbb (just an example). Why should I need to use mod rewrite (which only partially improves the situation)? And then why would I want to put a Google site search box on my site, when the site is not properly indexed?

I've found a number of spiders that are available that will crawl my site better than google. With some fine tuning, I have a much more complete index than google. And with some login, I hope to have a better weighting algorithm as well.

Anyone else having the same problems?

Offline takeshiro

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 13
  • Posts: 357
  • Karma: 2
  • Gender: Male
    • JavaScript Editor
Re: Google spider not good enough
« Reply #1 on: 11-02-2011, 09:48:53 »
yup, I agree with your stuff.


Related Topics

  Subject / Started by Replies Last post
8 Replies
Last post 09-02-2012, 03:38:22
by mariogames
3 Replies
Last post 08-12-2011, 04:32:41
by C.Rebecca
0 Replies
Last post 03-01-2012, 15:54:05
by shamanglove
12 Replies
Last post 11-27-2017, 10:09:36
by jeffjohnson
8 Replies
Last post 09-07-2016, 00:12:05
by TomClarke