Crawl conendrum

Author Topic: Crawl conendrum  (Read 1781 times)

Offline nicocoTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 1
  • Karma: 0
Crawl conendrum
« on: 05-08-2017, 07:53:32 »
Hi All

I am working on a student project and have been asked to develop a Local SEO strategy for a restaurant chain in the UK.
I was assigned a particular site and thought i would start with a screaming frog crawl to assess their current situation.
The crawl works well however it doesn't seem to pick up on each individual restaurant location such as
Bizarrely the URL does show up in google. I am not sure why this is happening. Can anyone help?

Many thanks

Offline steffidsouza46

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 5
  • Posts: 161
  • Karma: 0
  • Gender: Female
    • Corporate Day Care
Re: Crawl conendrum
« Reply #1 on: 05-12-2017, 10:31:30 »
A bot downloads the HTML document from your site, and parses out the interesting bits. This is how Google can crawl hundreds of millions of pages across the Internet. The process is very light and very repeatable.
This would allow the team to “crawl the program design” space in order to find the most effective implementation strategies.

Offline deekumar0203

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 7
  • Posts: 106
  • Karma: 0
Re: Crawl conendrum
« Reply #2 on: 05-19-2017, 23:48:44 »
Technically, you can scrape Quora's pages normally, however, legally, Quora does not allow this. Traditionally, you can get to know which pages a website allows for crawling/scraping through its robots.txt file

As you can see, thats a very big list of "Disallow"s.

So what does that mean if you still insist on doing so? Well, in most cases, they will be able to detect your crawler and cut it down. Or else they will haunt you down for infringement


Related Topics

  Subject / Started by Replies Last post
22 Replies
Last post 09-17-2012, 06:29:46
by thelazytrader
22 Replies
Last post 07-29-2022, 06:29:29
by Neha Pande
7 Replies
Last post 04-06-2017, 02:59:51
by sarahmaxwell789
0 Replies
Last post 10-17-2017, 02:34:40
by shabrishn
11 Replies
Last post 02-02-2021, 04:45:44
by marksmith112