What are Spiders, Robots and Crawlers ?

Author Topic: What are Spiders, Robots and Crawlers ?  (Read 1569 times)

Offline JoyadelfinTopic starter

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 5
  • Posts: 67
  • Karma: 0
    • Muslim Mantra For Love
What are Spiders, Robots and Crawlers ?
« on: 08-19-2015, 06:12:24 »
hello friends,


I want to know that What are Spiders, Robots and Crawlers and what are their functions? explain friends..


Offline Dewlance

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 2
  • Posts: 48
  • Karma: 0
    • Dewlance
Re: What are Spiders, Robots and Crawlers ?
« Reply #1 on: 08-22-2015, 07:36:25 »
I don't know to much but crawlers are programs which can copy our website text or full website and index site to their database.

Spiders are also a computer program but If you use common sense then you can easily understand what is meaning of Robots.
Cheap Windows VPS Hosting from Dewlance (Multiple Location)

Offline seoroy

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 7
  • Posts: 87
  • Karma: 0
  • Gender: Female
    • Wedding Planners in Delhi NCR
Re: What are Spiders, Robots and Crawlers ?
« Reply #2 on: 11-15-2015, 23:43:35 »
Spider - The browsers are like a program and to download the web page.


Robots - It had automated computer program can visit websites. It will guided by search engine algorithms It can combine the tasks of crawler & spider helpful of the indexing the web pages and through the search engines.

Crawler – The program is automatically to follow the links are web page.

Offline RH-Calvin

  • Trade Count: (0)
  • Hero Member
  • *****
  • Thank You 86
  • Posts: 1214
  • Karma: 0
  • Gender: Male
    • Cheap VPS Hosting
Re: What are Spiders, Robots and Crawlers ?
« Reply #3 on: 11-16-2015, 23:06:27 »
They are all the same and they are search engine programs that contain instructions to perform certain tasks online. They are mainly responsible to read through webpage sources and provide cache certificate to various webpages after successful crawling.

Offline seoroy

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 7
  • Posts: 87
  • Karma: 0
  • Gender: Female
    • Wedding Planners in Delhi NCR
Re: What are Spiders, Robots and Crawlers ?
« Reply #4 on: 11-23-2015, 03:04:50 »
Spider & crawler A bug is a program that visits Web destinations and peruses their pages and other data with a specific end goal to make passages for a web crawler record. The real web search tools on the Web all have such a project, which is otherwise called a "crawler" or a "bot." 

robots.txt It is extraordinary when web indexes as often as possible visit your website and record your substance however regularly there are situations when indexing parts of your online substance is not what you need. Case in point, on the off chance that you have two adaptations of a page , you'd rather have the printing adaptation prohibited from slithering, else you hazard being forced a copy content punishment.


 

Related Topics

  Subject / Started by Replies Last post
4 Replies
1648 Views
Last post 05-31-2010, 03:26:45
by coolman
5 Replies
1204 Views
Last post 11-05-2011, 00:01:18
by Vinil
4 Replies
1787 Views
Last post 01-19-2012, 22:54:39
by cpaoutsourcing
3 Replies
1288 Views
Last post 08-23-2016, 06:48:00
by TomClarke
7 Replies
1936 Views
Last post 04-06-2017, 02:59:51
by sarahmaxwell789