Learn How to solve robots.txt error (5xx error)

Author Topic: Learn How to solve robots.txt error (5xx error)  (Read 3783 times)

Offline akki5577Topic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 21
  • Karma: 0
  • Gender: Male
  • I am a positive persion
Learn How to solve robots.txt error (5xx error)
« on: 12-29-2017, 00:34:03 »
Hey guys, if you find 5xx (unreachable) error in your webmaster tool then do not worry I got a solution for that.
To solve error, You just need to contact your web hosting service provider and ask them for any issue. If you do not find any issue then login to your website’s cpanel and open file manager. Then you need to find robots.txt file in public_html section

If you do not find the file then you must upload a new file. Please note that, If you’re using WordPress, you might see a robots.txt file when you go to yoursite.com/robots.txt, but you won’t be able to find it in public_html section. This is because WordPress creates a virtual robots.txt file. In this case you’ll need to create a new robots.txt file.

Note : Save your robots.txt file as “robots” in your computer otherwise it will become "robots.txt.txt"

1.   Once you create the file, open your cpanel OR login to your FTP account
2.   Now go to file manager and select “Public_html”
3.   Then click on “Upload” and upload the file
4.   Now login to your webmaster account
5.   After that, “Crawl” in the left-hand sidebar.
6.   Then select “robots.txt Tester.”
7.   Click “Test” , If the “Test” changes to “Allowed,” that means your robots.txt is valid.

Now wait for at least 4 days to clear 5xx (unreachable) error.


Offline Neel Patel

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 11
  • Posts: 154
  • Karma: 7
  • Gender: Male
    • Autofresh
Re: Learn How to solve robots.txt error (5xx error)
« Reply #1 on: 12-29-2017, 22:37:08 »
Till yet not faced 5xx errors. but this will be useful in future in case i face any of 5xx error.
Guys must read once. explained properly and easily understandable.  !-!

Hey guys, if you find 5xx (unreachable) error in your webmaster tool then do not worry I got a solution for that.
To solve error, You just need to contact your web hosting service provider and ask them for any issue. If you do not find any issue then login to your website’s cpanel and open file manager. Then you need to find robots.txt file in public_html section

If you do not find the file then you must upload a new file. Please note that, If you’re using WordPress, you might see a robots.txt file when you go to yoursite.com/robots.txt, but you won’t be able to find it in public_html section. This is because WordPress creates a virtual robots.txt file. In this case you’ll need to create a new robots.txt file.

Note : Save your robots.txt file as “robots” in your computer otherwise it will become "robots.txt.txt"

1.   Once you create the file, open your cpanel OR login to your FTP account
2.   Now go to file manager and select “Public_html”
3.   Then click on “Upload” and upload the file
4.   Now login to your webmaster account
5.   After that, “Crawl” in the left-hand sidebar.
6.   Then select “robots.txt Tester.”
7.   Click “Test” , If the “Test” changes to “Allowed,” that means your robots.txt is valid.

Now wait for at least 4 days to clear 5xx (unreachable) error.

Offline kavyasharma561

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 13
  • Posts: 144
  • Karma: 0
Re: Learn How to solve robots.txt error (5xx error)
« Reply #2 on: 01-03-2018, 02:36:36 »
The robots.txt Tester tool shows you whether your robots.txt file blocks Google web crawlers from specific URLs on your site. For example, you can use this tool to test whether the Googlebot-Image crawler can crawl the URL of an image you wish to block from Google Image Search.

Offline evejones

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 5
  • Posts: 70
  • Karma: 0
  • Gender: Female
    • Fisher & Associates P.C.
Re: Learn How to solve robots.txt error (5xx error)
« Reply #3 on: 01-03-2018, 21:50:01 »
Hey guys, if you find 5xx (unreachable) error in your webmaster tool then do not worry I got a solution for that.
To solve error, You just need to contact your web hosting service provider and ask them for any issue. If you do not find any issue then login to your website’s cpanel and open file manager. Then you need to find robots.txt file in public_html section

If you do not find the file then you must upload a new file. Please note that, If you’re using WordPress, you might see a robots.txt file when you go to yoursite.com/robots.txt, but you won’t be able to find it in public_html section. This is because WordPress creates a virtual robots.txt file. In this case you’ll need to create a new robots.txt file.

Note : Save your robots.txt file as “robots” in your computer otherwise it will become "robots.txt.txt"

1.   Once you create the file, open your cpanel OR login to your FTP account
2.   Now go to file manager and select “Public_html”
3.   Then click on “Upload” and upload the file
4.   Now login to your webmaster account
5.   After that, “Crawl” in the left-hand sidebar.
6.   Then select “robots.txt Tester.”
7.   Click “Test” , If the “Test” changes to “Allowed,” that means your robots.txt is valid.

Now wait for at least 4 days to clear 5xx (unreachable) error.

Errors are really disgusting and these steps fluently describe how to overcome them with robots.txt. Thanks a lot for this!

Offline richardmsmith

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 1
  • Posts: 230
  • Karma: 0
  • Gender: Male
    • Trucking App Solution
Re: Learn How to solve robots.txt error (5xx error)
« Reply #4 on: 07-11-2018, 04:30:09 »
Robots file is for stopping the search engine to crawling the website. It is not showing the error. If you find this one than may be its another problem.



Offline organictextiles

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 55
  • Karma: 0
    • https://www.organictextiles.com/
Re: Learn How to solve robots.txt error (5xx error)
« Reply #5 on: 07-11-2018, 05:37:57 »
Hi

The robots.txt Tester tool shows you whether your robots.txt file blocks Google web crawlers from specific URLs on your site. For example, you can use this tool to test whether the Googlebot-Image crawler can crawl the URL of an image you wish to block from Google Image Search.
https://www.organictextiles.com/

 

Related Topics

  Subject / Started by Replies Last post
3 Replies
2624 Views
Last post 01-29-2015, 01:20:37
by kumar29
2 Replies
1034 Views
Last post 10-18-2016, 00:04:24
by RH-Calvin
16 Replies
4566 Views
Last post 09-11-2018, 22:07:28
by pitterjhon44
1 Replies
1215 Views
Last post 01-10-2017, 03:05:38
by knila
1 Replies
313 Views
Last post 09-29-2018, 19:25:40
by marrysmith101