Is it necessary to use Robots.txt?

Author Topic: Is it necessary to use Robots.txt?  (Read 1741 times)

Offline AliceFowell89Topic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 37
  • Karma: 0
    • Video Conferencing Australia
Is it necessary to use Robots.txt?
« on: 05-18-2016, 05:25:38 »
Hi,

My question is, Is it necessary to use Robots.txt?


Offline SEO.Ninja

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 106
  • Karma: 0
  • Gender: Female
    • SEO Ninja
Re: Is it necessary to use Robots.txt?
« Reply #1 on: 05-18-2016, 05:47:15 »
Robots.txt is also called Robot Exclusion Protocol (REP).
It is a text file webmaster creat to instruct robots how to crawl and index pages on their websites.

Offline RH-Calvin

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 52
  • Posts: 1018
  • Karma: -1
  • Gender: Male
    • Cheap VPS Hosting
Re: Is it necessary to use Robots.txt?
« Reply #2 on: 05-19-2016, 01:47:09 »
Robots.txt is text file inserted into the website to contain instructions for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. They help to control the search engine robots in your website.

Offline TomClarke

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 17
  • Posts: 896
  • Karma: 0
  • Gender: Male
    • Custom Writing Service & Content Marketplace
Re: Is it necessary to use Robots.txt?
« Reply #3 on: 05-19-2016, 03:01:39 »
I think that your WordPress website will be indexed by search engines even without that file present.  WordPress in itself already contains a virtual robots.txt. One thing that you should be aware of is that obedience to robots.txt cannot be enforced. The file will be recognized and respected by major search engines, but malicious crawlers and low-quality search crawlers might disregard it completely.
Your ideas, our writings - Writology.com

Offline arindamdutta16

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 5
  • Posts: 339
  • Karma: 0
  • Gender: Male
  • WIN the SEO Battle!!!
    • Web Design Services
Re: Is it necessary to use Robots.txt?
« Reply #4 on: 05-19-2016, 04:38:23 »
The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.


Offline stompinhere

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 3
  • Karma: 0
Re: Is it necessary to use Robots.txt?
« Reply #5 on: 05-19-2016, 05:12:20 »
My sites have not had a robots.txt file since they started 9 years ago and it has never affected their ranking, what has is googles algorithms and its a constant battle to get penalised sites back up the SERP's and ranking!

Offline Roy Milson

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 3
  • Posts: 107
  • Karma: 0
Re: Is it necessary to use Robots.txt?
« Reply #6 on: 05-20-2016, 00:16:52 »
It is not important use robots. txt file but it is advisable to use it, if there is any page on your websites which you don't want search engine to crawl. Robots.txt also improves better indexing for the website.

Offline MaryGreen

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 5
  • Posts: 429
  • Karma: 1
  • Gender: Female
    • exclusive-paper.com
Re: Is it necessary to use Robots.txt?
« Reply #7 on: 05-24-2016, 01:30:21 »
I think, you can use it! Robots.txt file helps search engine robots to direct which part to crawl and which part to avoid. Of course, the best platform for Robots.txt is a WordPress

Offline abdul ghany

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 29
  • Karma: 0
  • Gender: Male
    • Free classified india
Re: Is it necessary to use Robots.txt?
« Reply #8 on: 05-25-2016, 02:04:11 »
Depends on you,

Web site owners use the robots.txt file to give instructions about their site to web robots, to dont crawl specific page like admin pannels or high secure pages


Offline Nimit.Suri

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 45
  • Karma: 0
  • Gender: Male
    • IT Solutions |IT Outsourcing and Consulting Services - Synergy Technology Services
Re: Is it necessary to use Robots.txt?
« Reply #9 on: 05-25-2016, 07:19:33 »
Hello
Robot.txt is very important for your seo and for your website on the whole. Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to.robots.It is  a very important file if you want to have a good ranking on search engines

I found this information online .Hope it helps you to understand the importance of robot.txt
Reasons why Robots.txt file is useful:

    It lets you to block search engine spiders from crawling any duplicate pages on your website.
    You can block search engines to index your internal search results pages.
    If you don’t want search engines to index certain areas of your website or a whole website.
    You can block search engines to index certain files on your website (images, PDFs, etc.)
    You can notify search engines where your sitemap file is located

 

Related Topics

  Subject / Started by Replies Last post
24 Replies
3321 Views
Last post 05-10-2012, 01:38:00
by ring2012
32 Replies
7520 Views
Last post 04-24-2016, 20:52:14
by nguyenmailan
10 Replies
1910 Views
Last post 11-29-2012, 04:09:32
by seocyrusjpr
17 Replies
5486 Views
Last post 04-19-2014, 00:47:04
by ComoliWesta
3 Replies
1724 Views
Last post 09-06-2016, 01:24:52
by TomClarke