Crawl conendrum

Author Topic: Crawl conendrum  (Read 949 times)

Offline nicocoTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 1
  • Karma: 0
Crawl conendrum
« on: 05-08-2017, 07:53:32 »
Hi All

I am working on a student project and have been asked to develop a Local SEO strategy for a restaurant chain in the UK.
I was assigned a particular site http://www.caferouge.com/ and thought i would start with a screaming frog crawl to assess their current situation.
The crawl works well however it doesn't seem to pick up on each individual restaurant location such as http://www.caferouge.com/locations/kew-bridge/book
Bizarrely the URL does show up in google. I am not sure why this is happening. Can anyone help?

Many thanks


Offline steffidsouza46

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 4
  • Posts: 175
  • Karma: 0
  • Gender: Female
    • Corporate Day Care
Re: Crawl conendrum
« Reply #1 on: 05-12-2017, 10:31:30 »
A bot downloads the HTML document from your site, and parses out the interesting bits. This is how Google can crawl hundreds of millions of pages across the Internet. The process is very light and very repeatable.
This would allow the team to “crawl the program design” space in order to find the most effective implementation strategies.

Offline deekumar0203

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 7
  • Posts: 126
  • Karma: 0
Re: Crawl conendrum
« Reply #2 on: 05-19-2017, 23:48:44 »
Technically, you can scrape Quora's pages normally, however, legally, Quora does not allow this. Traditionally, you can get to know which pages a website allows for crawling/scraping through its robots.txt file

As you can see, thats a very big list of "Disallow"s.

So what does that mean if you still insist on doing so? Well, in most cases, they will be able to detect your crawler and cut it down. Or else they will haunt you down for infringement

 

Related Topics

  Subject / Started by Replies Last post
22 Replies
4923 Views
Last post 09-17-2012, 06:29:46
by thelazytrader
6 Replies
3211 Views
Last post 12-01-2014, 23:54:34
by AvniShergill
9 Replies
4794 Views
Last post 05-17-2017, 00:50:13
by deekumar0203
14 Replies
3797 Views
Last post 07-31-2017, 02:35:34
by Hitesh Patel
0 Replies
49 Views
Last post 10-17-2017, 02:34:40
by shabrishn