Google spider not good enough

[manvbf]

Well, I feel like I've been forced to take matters into my own hands, and build my own site search. I've tried quite a few different things, from site maps to mod rewrite. It seems that google decides how many pages it wants to index on a site based on its own algorithm that determines the importance of a site. That's fine for Google, but the users of my site suffer. There are too many "supplemental" results and unindexed pages.

Take phpbb (just an example). Why should I need to use mod rewrite (which only partially improves the situation)? And then why would I want to put a Google site search box on my site, when the site is not properly indexed?

I've found a number of spiders that are available that will crawl my site better than google. With some fine tuning, I have a much more complete index than google. And with some login, I hope to have a better weighting algorithm as well.

Anyone else having the same problems?

[takeshiro]

yup, I agree with your stuff.