Crawl Error

Author Topic: Crawl Error  (Read 4655 times)

Offline logosoukigniteTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 17
  • Karma: 0
  • Ecommerce solutions for small businesses
    • Ecommerce Solutions for Small Business - IGNITE
Crawl Error
« on: 06-02-2020, 09:29:47 »
Crawl Errors page provides details about the URLs that Google could not successfully crawl or that returned an HTTP error code.


Offline brill_mindz

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 4
  • Posts: 75
  • Karma: 0
    • Best Mobile App development company in Saudi Arabia
Re: Crawl Error
« Reply #1 on: 06-03-2020, 02:33:10 »
Crawl errors occur when a search engine tries to reach a webpage but fails to do so. There are two types of crawl errors:

Site errors (which mean your entire site can’t be crawled)
URL errors (which relate to one specific URL per error)
Site Errors in Google Search Console

DNS Errors
A DNS (Domain Name System) error means that a search engine can’t communicate with your server either because the server is down or because there's an issue with the DNS routing to your domain. Usually, this is a temporary issue.

Server Errors
When you see this type of error for your URLs, it means that Googlebot wasn’t able to access your URL, the request timed out, or your site was busy. This can also mean that your website has so many visitors that the server just couldn’t handle all the requests. Most of these errors are defined as 5xx status codes.

Robots failure
Before Googlebot crawls your website, it tries to crawl your robots.txt file to see if there are any pages you’d rather not have indexed. If that bot can’t reach the robots.txt file (i.e. robots.txt file doesn’t return a 200 or 404 HTTP status code), Google will postpone the crawl rather than risk crawling URLs that you don’t want to be crawled. That’s why you always want to make sure the robots.txt file is available.

URL Errors in Google Search Console

Common URL Errors
Common URL errors occur when a search engine tries to crawl a specific web page. Among them might be an occasional DNS error or server error for that specific URL.

Mobile-Only URL Errors
This type of error refers to the crawl failure that occurs on smart devices. If your website is responsive, mobile-only URL errors are unlikely to appear.

The most usual mobile-only URL errors are faulty redirects. Some websites use different URLs for desktop and smartphone users. A faulty redirect occurs when a desktop page incorrectly redirects mobile users to a page not relevant to their query.

Offline Nizam18

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 60
  • Karma: 0
  • Gender: Male
    • Digital Transformation Company
Re: Crawl Error
« Reply #2 on: 06-15-2020, 11:45:54 »
Crawl errors occur when a search engine tries to reach a page on your website but fails at it. Let’s shed some more light on crawling first. Crawling is the process where a search engine tries to visit every page of your website via a bot. A search engine bot finds a link to your website and starts to find all your public pages from there. The bot crawls the pages and indexes all the contents for use in Google, plus adds all the links on these pages to the pile of pages it still has to crawl. Your main goal as a website owner is to make sure the search engine bot can get to all pages on the site. Failing this process returns what we call crawl errors.
Nizam Ali,
Marketing Executive - Digital Transformation Company.

Offline eSage IT

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 1
  • Posts: 88
  • Karma: 0
  • Gender: Male
    • eCommerce Marketing Agency
Re: Crawl Error
« Reply #3 on: 06-16-2020, 05:13:00 »
Crawl errors occur when a search engine tries to reach a webpage on a website but fails to do so. Crawl errors are of different types such as Site errors, URL errors, and more.

Offline Yuga

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 86
  • Karma: 0
    • Codism LLC
Re: Crawl Error
« Reply #4 on: 12-07-2020, 04:58:37 »
Crawl errors occur when a search engine tries to reach a page on your website but fails at it. ... Your goal is to make sure that every link on your website leads to an actual page. That might be via a 301 redirect, but the page at the very end of that link should always return a 200 OK server response


Offline danielbora123

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 54
  • Karma: 0
  • Gender: Male
  • We provide IT services
    • Zworthkey
Re: Crawl Error
« Reply #5 on: 12-14-2020, 00:38:03 »
The Fix

The fix here is very similar to the first one. Here are a few tips:

1-Set your main home page URL to be http://www.mysite.com. 301 redirect everything else to this URL since it is the most basic (If your preferred URL style is non-www, then do http://mysite.com)

2-If you have to set up different versions for tracking purposes or some other reason, set a canonical tag to show the search engines which version you want showing up in the search results.

3-Make sure when you build links to your home page that you link to the correct version.
ZWorthKey

Offline John - Smith

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 15
  • Posts: 305
  • Karma: 6
  • Gender: Male
    • online-paystubs
Re: Crawl Error
« Reply #6 on: 12-25-2020, 23:20:33 »
I guess you guys covered a to z about crawl errors! Any doubts left @ op?

Offline ChrisJames661

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 25
  • Karma: 0
Re: Crawl Error
« Reply #7 on: 12-30-2020, 06:36:19 »
Fix your crawl errors
Very specific URL errors
URL errors
Site errors

Offline mariajonas

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 3
  • Posts: 99
  • Karma: 0
    • Free Legal Document Creator & Form Templates Generator
Re: Crawl Error
« Reply #8 on: 01-01-2021, 01:43:31 »
@OP, what is the purpose of starting this thread?
✦Document✦Forms✦Templates✦Contracts
legal document generator | general release of liability form


Offline Digital Marketing Ireland

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 1
  • Karma: 0
  • SEO Company
    • Digital Marketing Ireland
Re: Crawl Error
« Reply #9 on: 01-03-2021, 04:18:51 »
You need to understand if the crawl error is happening because of your...
- server
- site or page
- robots.txt

If it is server, several issues might be happening: denying connection from an IP address (firewall), PHP or other language problem, Apache server problem. This type of problem will likely not let you even see the website/page.

If the problem is the website (or the CMS / Wordpress), then you need to check if you have internal errors on the CMS or if your Permalinks are working (Permalinks problem will also refer to server errors: see PHP mod rewrite or .htaccess).

Check your robots.txt  to see if it's allowing crawlers. This may also refer to wordpress / CMS not allowing search engines to crawl the website, especially if you have SEO plugins.

If you can't find the root of the problem and it's happening only for one or a few pages, you can try 301 redirects to redirect the problematic page to a working one.
Vronscki.com is an SEO Agency in Ireland helping companies with Search Engine Optimization, Local SEO, Social Media, Pay per Click Ads and Web Design.

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
1087 Views
Last post 08-27-2016, 11:37:14
by aghosh
3 Replies
4026 Views
Last post 11-29-2016, 07:48:21
by damponting44
0 Replies
1012 Views
Last post 10-17-2017, 02:34:40
by shabrishn
5 Replies
5511 Views
Last post 07-11-2018, 05:37:57
by organictextiles
1 Replies
2513 Views
Last post 09-29-2018, 19:25:40
by marrysmith101