What is Robots.txt?

Author Topic: What is Robots.txt?  (Read 4196 times)

Offline sreelavanyaTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 38
  • Karma: 2
What is Robots.txt?
« on: 03-26-2012, 00:46:35 »
 !-!What is Robots.txt?
 
As the file name and extension suggest, a robots.txt file is a text file that gives instructions to search engine robots. A search engine robot, also sometimes called a bot, spider, or web crawler, is a program that systematically browses virtually all the websites on the Internet. The program gathers the most recent data and feeds this data back into to the search engine algorithm. The search engine algorithm is a mathematical formula that determines where and how to categorize various websites.
 
The robots.txt file is like your website's helpful tour guide for the search robot. When a robot crawls your site, the robots.txt file is the first place it goes. Inside the text file, instructions to the robot are stored that tell the program which areas of the site shouldn't be crawled at all.

.......................
plc training in india|plc training in india


Offline bryanbell

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 7
  • Posts: 107
  • Karma: 2
  • Gender: Male
Re: What is Robots.txt?
« Reply #1 on: 03-26-2012, 23:01:49 »
Generally robots.txt file is use to inform search engine spider because whenever crawler come on website they will find this file which is located in rood directory of website. From this file crawler will get information which page is indexed and ignored, it is text file.

Offline shawn88

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 29
  • Karma: 2
  • Gender: Male
    • Business Pro Designs
Re: What is Robots.txt?
« Reply #2 on: 03-27-2012, 00:47:33 »
hello,
There is a hidden, relentless force that permeates the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

regards
shawn

Offline AndersonP

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 1
  • Posts: 10
  • Karma: 0
  • Gender: Male
    • Perceptive Mobile Apps
Re: What is Robots.txt?
« Reply #3 on: 03-28-2012, 00:18:40 »
Hello sreelavanya..,

Thanks for Sharing Nice information about Robots.txt. As I read somewhere - Robots.txt tells the robot that it should not visit any pages on the site. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Offline axix245

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 8
  • Posts: 43
  • Karma: 1
    • Motorcycle Clothing
Re: What is Robots.txt?
« Reply #4 on: 03-28-2012, 03:29:03 »
Actually companies used that file "Robots.txt" for hide private page to search engine.due to this file search engine will unable to read that page...


Offline John23

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 9
  • Posts: 119
  • Karma: 3
  • Gender: Male
Re: What is Robots.txt?
« Reply #5 on: 04-02-2012, 04:21:22 »
Basically, robot.txt is crawler of your website and they give your website data, update to search engine. If you don't want to index any page from search engine then you can disallow that page by robot.txt.

Offline puviyamilla

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 7
  • Karma: 0
Re: What is Robots.txt?
« Reply #6 on: 04-09-2012, 04:26:34 »
 Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from newbielink:http://www.xnynz.com/ [nonactive]

Offline tianamartin

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 9
  • Posts: 42
  • Karma: 0
  • Gender: Female
    • Tech Lido
Re: What is Robots.txt?
« Reply #7 on: 04-09-2012, 05:29:17 »
Basically in short this this a code to hide someones personal information. or the thing which they do not want to disclose it publicly.companies used that file "Robots.txt" to hide secret pages or information.due Robot.txt file file search engine can not access that page.

Offline kenrylesso

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 41
  • Karma: 0
    • Hotels johor bahru
Re: What is Robots.txt?
« Reply #8 on: 04-09-2012, 19:08:50 »
This is a notepad file which helps to remove our web pages from Google index or if we want not to crawl web pages from Google spider then we place these links in robots.txt file.


Offline Smith Jones

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 39
  • Posts: 342
  • Karma: 5
  • Gender: Male
Re: What is Robots.txt?
« Reply #9 on: 04-10-2012, 05:04:48 »
Hello friends,

Robots. TXT is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Thanks for sharing.
Smith Jones

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1563 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1284 Views
Last post 08-23-2016, 06:48:00
by TomClarke
11 Replies
3482 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
1285 Views
Last post 12-28-2016, 05:14:34
by rustyblogger
4 Replies
2135 Views
Last post 12-27-2016, 03:47:18
by damponting44