URLextractor - Information Gathering and Website Reconnaissance

✨ deeznutz

✨ Master ✨
Staff member
May 15, 2017

Information gathering & website reconnaissance

  • Colorex: put colors to the ouput pip install colorex and use it like ./extractor http://www.hackthissite.org/| colorex -g "INFO" -r "ALERT"
  • Tldextract: is used by dnsenumeration function pip install tldextract
  • IP and hosting info like city and country (using FreegeoIP)
  • DNS servers (using dig)
  • ASN, Network range, ISP name (using RISwhois)
  • Load balancer test
  • Whois for abuse mail (using Spamcop)
  • PAC (Proxy Auto Configuration) file
  • Compares hashes to diff code
  • robots.txt (recursively looking for hidden stuff)
  • Source code (looking for passwords and users)
  • External links (frames from other websites)
  • Directory FUZZ (like Dirbuster and Wfuzz - using Dirbuster) directory list)
  • URLvoid API - checks Google page rank, Alexa rank and possible blacklists
  • Provides useful links at other websites to correlate with IP/ASN
  • Option to open ALL results in browser at the end
Changelog to version 0.2.0:
  • [Fix] Changed GeoIP from freegeoip to ip-api
  • [Fix/Improvement] Remove duplicates from robots.txt
  • [Improvement] Better whois abuse contacts (abuse.net)
  • [Improvement] Top passwords collection added to sourcecode checking
  • [New feature] Firt run verification to install dependencies if need
  • [New feature] Log file
  • [New feature] Check for hostname on log file
  • [New feature] Check if hostname is listed on Spamaus Domain Blacklist
  • [New feature] Run a quick dnsenumeration with common server names
Changelog to version 0.1.9:
  • Abuse mail using lynx istead of curl
  • Target server name parsing fixed
  • More verbose about HTTP codes and directory discovery
  • MD5 collection for IP fixed
  • Links found now show unique URLs from array
  • [New feature] Google results
  • [New feature] Bing IP check for other hosts/vhosts
  • [New feature] Opened ports from Shodan
  • [New feature] VirusTotal information about IP
  • [New feature] Alexa Rank information about $TARGET_HOST
Tested on Kali light mini AND OSX 10.11.3 with brew
sudo apt-get install bc curl dnsutils libxml2-utils whois md5sha1sum lynx openssl -y
Download URLextractor
Top Bottom