Why robert.txt file is important to Website ?

Author Topic: Why robert.txt file is important to Website ?  (Read 507 times)

Offline jasminekour12Topic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 34
  • Karma: 0
  • Gender: Female
  • I am just a simple girl, with simple wishes, simpl
    • Vashikaran mantra
Why robert.txt file is important to Website ?
« on: 10-11-2016, 01:31:23 »
Hello Friends,


  I want to know that Why  robert.txt file is important to Website ? tell me guys,,


Offline Victorvictories

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 41
  • Karma: 0
Re: Why robert.txt file is important to Website ?
« Reply #1 on: 10-11-2016, 03:37:37 »
Robots file is beneficial to block the particular pages or directory which the webmaster doesn’t want to be crawled by the crawler. Robots file is used to instruct which Pages will be crawled or which not.

Offline Intuz

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 38
  • Karma: 0
    • Intuz - Custom Mobile app development & Cloud Solutions
Re: Why robert.txt file is important to Website ?
« Reply #2 on: 10-11-2016, 06:47:22 »
Yes, robots.txt file of website is important because the robots.txt file is a simple text file placed on your web server which tells webcrawlers like Googlebot if they should access a file or not. The robots.txt file controls how search engine spiders see and interact with your webpages

Offline AmarInfotech

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 30
  • Karma: 0
  • Gender: Male
  • We knows Whats better
    • Web Development Company in INDIA
Re: Why robert.txt file is important to Website ?
« Reply #3 on: 10-11-2016, 22:59:50 »
Robots.txt helps to block the web pages which has sensitive information of our website. It may be a admin panel or other secrate information of the business. Robots.txt instructs search engine that not crawl this pages. By this, we can keep our data secure.
Amarinfotech is an IT development company that offers SEO services in Sydney and helps to a webmaster to solve their issue.

Offline alex.thomson

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 0
  • Posts: 65
  • Karma: 0
Re: Why robert.txt file is important to Website ?
« Reply #4 on: 10-12-2016, 02:23:53 »
You might be surprised to hear that one small text file, known as robots.txt, could be the downfall of your website.  If you get the file wrong you could end up telling search engine robots not to crawl your site, meaning your web pages won’t appear in the search results.  Therefore, it’s important that you understand the purpose of a robots.txt file and learn how to check you’re using it correctly.


 

Related Topics

  Subject / Started by Replies Last post
2 Replies
1742 Views
Last post 01-08-2013, 11:40:55
by localnumberone2
32 Replies
7272 Views
Last post 10-14-2014, 21:29:09
by Alawi
0 Replies
776 Views
Last post 06-27-2013, 06:55:43
by David-Luz
13 Replies
4271 Views
Last post 08-12-2016, 01:26:32
by TomClarke
0 Replies
489 Views
Last post 04-02-2016, 04:57:02
by chinmay.sahoo