Crawl conendrum

Author Topic: Crawl conendrum  (Read 1046 times)

Offline nicocoTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 1
  • Karma: 0
Crawl conendrum
« on: 05-08-2017, 07:53:32 »
Hi All

I am working on a student project and have been asked to develop a Local SEO strategy for a restaurant chain in the UK.
I was assigned a particular site http://www.caferouge.com/ and thought i would start with a screaming frog crawl to assess their current situation.
The crawl works well however it doesn't seem to pick up on each individual restaurant location such as http://www.caferouge.com/locations/kew-bridge/book
Bizarrely the URL does show up in google. I am not sure why this is happening. Can anyone help?

Many thanks


Offline steffidsouza46

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 4
  • Posts: 167
  • Karma: 0
  • Gender: Female
    • Corporate Day Care
Re: Crawl conendrum
« Reply #1 on: 05-12-2017, 10:31:30 »
A bot downloads the HTML document from your site, and parses out the interesting bits. This is how Google can crawl hundreds of millions of pages across the Internet. The process is very light and very repeatable.
This would allow the team to “crawl the program design” space in order to find the most effective implementation strategies.

Offline deekumar0203

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 7
  • Posts: 110
  • Karma: 0
Re: Crawl conendrum
« Reply #2 on: 05-19-2017, 23:48:44 »
Technically, you can scrape Quora's pages normally, however, legally, Quora does not allow this. Traditionally, you can get to know which pages a website allows for crawling/scraping through its robots.txt file

As you can see, thats a very big list of "Disallow"s.

So what does that mean if you still insist on doing so? Well, in most cases, they will be able to detect your crawler and cut it down. Or else they will haunt you down for infringement

 

Related Topics

  Subject / Started by Replies Last post
22 Replies
5114 Views
Last post 09-17-2012, 06:29:46
by thelazytrader
17 Replies
4290 Views
Last post 01-24-2013, 20:57:54
by quain88
6 Replies
3330 Views
Last post 12-01-2014, 23:54:34
by AvniShergill
7 Replies
2081 Views
Last post 04-06-2017, 02:59:51
by sarahmaxwell789
0 Replies
179 Views
Last post 10-17-2017, 02:34:40
by shabrishn