Introduction
Link crawling is the process of capturing all the pages (URLs/links) present on a website. It helps us know how many and which pages are there in our website. Site owners can also crosscheck whether these page are legitimate or not.
ESDS VTMScan (https://www.esds.co.in/vtmscan) Technique
We crawl all the pages which are present on a website by actually visiting the pages and links which are present in the view source of the page. In crawling we capture URLs/Pages by following methods:
Page Visit: We visit each and every page and links present on them via this recursive process. If we get 200, 403 response then we add this in URL file.
Admin Buster: We append admin page keywords such as index.EXT, admin.EXT, login.htm, login/, login.EXT, login/login.EXT, adm/, siteadmin.EXT, etc. After scanning the URL for e.g. http://demotest.com/admin.php if we get 200, 403 response then we add this in URL file.
Directory Buster: We append admin page keywords such as index, images, wimages, imgs, img, iconset, icons, home, etc. After scanning the URL for e.g. http://demotest.com/admin if we get 200, 403 response then we add this in URL file.
Keep sharing such useful threads with community. !-!