Author Topic: Robots.txt  (Read 1601 times)

Offline Fadia SheetalTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 48
  • Karma: 0
  • Gender: Female
    • Buy Spoken English Books Online and Personality Development in Surat
« on: 08-05-2015, 06:06:30 »
What are the different methods to control Robots.txt

Offline James_Tarrier

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 3
  • Karma: 0
    • Send Money Home
Re: Robots.txt
« Reply #1 on: 08-05-2015, 06:40:37 »
When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.
newbielink:http://" [nonactive]

Offline Prabha Yadav

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 38
  • Karma: 0
    • Dev Technosys
Re: Robots.txt
« Reply #2 on: 08-05-2015, 07:25:06 »
    First of all, we used Robots.txt to control search engine crawling. It can be used to bound search engine for crawling a perticular web page or a directory or the whole site and vise versa. And there is no method exist to control robots.txt file.

For more details about Robots.txt, you can read this post: or

If you want to ask or any other assistance regarding SEO services, you can contact us.


Offline TomClarke

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 15
  • Posts: 896
  • Karma: 0
Re: Robots.txt
« Reply #3 on: 09-06-2016, 01:24:52 »
Here is how robots.txt can be used:
- for non-image files robots.txt  should only be used to control crawling traffic, typically because you don't want your server to be overwhelmed by Goolge's crawler or to waste crawl budget  crawling uimportant or similar pages on oyur site. You should not use robots.txt as a means to hide webpages from Google Search results. This is because other pages might point to your page, and your page could get indexed that way, avoiding the robots.txt file. If you want to block your page from search results, use another method such as password protection or noindex tags or directives;
- robots.txt does prevent image files from appearing in Google search results;
- you can use robots.txt  block resource files such as unimportant image, scriptm, or style files, if you think that pages loaded without these resources will not be significantly affected by the loss. However, if the absence of these resources make the page harder to understand for Google's crawler, you should not block them, or else Google won't do a good job of analyzing your pages that depend on those resources.


Related Topics

  Subject / Started by Replies Last post
24 Replies
Last post 05-10-2012, 01:38:00
by ring2012
33 Replies
Last post 04-24-2016, 20:52:14
by nguyenmailan
10 Replies
Last post 11-29-2012, 04:09:32
by seocyrusjpr
17 Replies
Last post 04-19-2014, 00:47:04
by ComoliWesta
13 Replies
Last post 10-05-2016, 06:39:48
by mukeshkr