Link Crawling

[manoharparakh]

Introduction
Link crawling is the process of capturing all the pages (URLs/links) present on a website. It helps us know how many and which pages are there in our website. Site owners can also crosscheck whether these page are legitimate or not.

ESDS VTMScan Technique
We crawl all the pages which are present on a website by actually visiting the pages and links which are present in the view source of the page. In crawling we capture URLs/Pages by following methods:

Page Visit: We visit each and every page and links present on them via this recursive process. If we get 200, 403 response then we add this in URL file.

Admin Buster: We append admin page keywords such as index.EXT, admin.EXT, login.htm, login/, login.EXT, login/login.EXT, adm/, siteadmin.EXT, etc. After scanning the URL for e.g. http://demotest.com/admin.php if we get 200, 403 response then we add this in URL file.

Directory Buster: We append admin page keywords such as index, images, wimages, imgs, img, iconset, icons, home, etc. After scanning the URL for e.g. http://demotest.com/admin if we get 200, 403 response then we add this in URL file.

[smartscraper]

Keep sharing such useful threads with community.