Robot.txt Access

Author Topic: Robot.txt Access  (Read 2590 times)

Offline Sap SolutionTopic starter

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 144
  • Karma: 0
Robot.txt Access
« on: 01-16-2018, 07:15:18 »
If a robot.txt file is not accessed by the Google , what can be problem to our website ?

Offline Chilli-Group

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 1
  • Posts: 6
  • Karma: 0
  • Gender: Male
  • Digital Marketing Company in Sunshine Coast
    • Digital Marketing Company in Sunshine Coast
Re: Robot.txt Access
« Reply #1 on: 01-16-2018, 23:22:15 »
The first thing a search engine spider like Googlebot looks at when it is visiting a page is the robots.txt file. It does this because it wants to know if it has permission to access that page or file. If the robots.txt file says it can enter, the search engine spider then continues on to the page files.

If you have instructions for a search engine robot, you must tell it those instructions. The way you do so is the robots.txt file.

Improper usage of the robots.txt file can hurt your ranking
The robots.txt file controls how search engine spiders see and interact with your webpages
This file is mentioned in several of the Google guidelines
This file, and the bots they interact with, are fundamental parts of how search engines work

If you need more support on SEO, We are here to help you. Please Google for Chilli Group Website or give us a Call: 07 5437 7788
newbielink: [nonactive] | newbielink: [nonactive] | [url=

Offline akki5577

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 21
  • Karma: 0
  • Gender: Male
  • I am a positive persion
Re: Robot.txt Access
« Reply #2 on: 01-17-2018, 23:22:13 »
Search engine will look at your robots.txt file and then crawls your site, robots.txt helps you to ignore any duplicate pages on your site.

If you do not want to get rank for certain pages like “Terms & Conditions”, “Register page”, “Login Page”, “disclaimer” page then you need to submit robots.txt file
You can also use “no index” tags.


Related Topics

  Subject / Started by Replies Last post
6 Replies
Last post 10-18-2019, 04:47:50
by ifixservices
18 Replies
Last post 08-22-2016, 04:05:22
by Harikrishna Patel
1 Replies
Last post 10-25-2017, 04:12:05
by damponting44
2 Replies
Last post 06-10-2021, 01:46:42
by saravanan28