OSINT Tool Guide & Demonstration: The Harvester

🛠 What the tool does

“theHarvester” is a simple to use, yet powerful tool designed to be used during the reconnaissance stage of a red team assessment or penetration test. It performs open source intelligence (OSINT) gathering to help determine a domain's external threat landscape. The tool gathers names, emails, IPs, subdomains, and URLs by using multiple public resources that include:

Passive modules -

• Baidu : Baidu search engine - www.baidu.com

• Bevigil : CloudSEK BeVigil scans mobile application for OSINT assets (Requires an API key, see below.) - https://bevigil.com/osint-api

• Bing : Microsoft search engine - https://www.bing.com

• Bingapi : Microsoft search engine, through the API (Requires an API key, see below.)

• Brave : Brave search engine - https://search.brave.com/

• Bufferoverun : (Requires an API key, see below.) https://tls.bufferover.run

• Censys : Censys search engine will use certificates searches to enumerate subdomains and gather emails
  (Requires an API key, see below.) https://censys.io

• Certspotter : Cert Spotter monitors Certificate Transparency logs - https://sslmate.com/certspotter/

• Criminalip : Specialized Cyber Threat Intelligence (CTI) search engine (Requires an API key, see below.) - https://www.criminalip.io

• Crtsh : Comodo Certificate search - https://crt.sh

• Duckduckgo : DuckDuckGo search engine - https://duckduckgo.com

• Fullhunt : Next-generation attack surface security platform (Requires an API key, see below.) - https://fullhunt.io

• github-code : GitHub code search engine (Requires a GitHub Personal Access Token, see below.) - www.github.com

• hackertarget : Online vulnerability scanners and network intelligence to help organizations - https://hackertarget.com

• hunter : Hunter search engine (Requires an API key, see below.) - https://hunter.io

• hunterhow : Internet search engines for security researchers (Requires an API key, see below.) - https://hunter.how

• intelx : Intelx search engine (Requires an API key, see below.) - http://intelx.io

• netlas : A Shodan or Censys competitor (Requires an API key, see below.) - https://app.netlas.io

• onyphe : Cyber defense search engine (Requires an API key, see below.) - https://www.onyphe.io/

• otx : AlienVault open threat exchange - https://otx.alienvault.com

• pentestTools : Cloud-based toolkit for offensive security testing, focused on web applications and network penetration
  testing (Requires an API key, see below.) - https://pentest-tools.com/

• projecDiscovery : We actively collect and maintain internet-wide assets data, to enhance research and analyse changes around
  DNS for better insights (Requires an API key, see below.) - https://chaos.projectdiscovery.io

• rapiddns : DNS query tool which make querying subdomains or sites of a same IP easy! https://rapiddns.io

• rocketreach : Access real-time verified personal/professional emails, phone numbers, and social media links (Requires an API key,
  see below.) - https://rocketreach.co

• securityTrails : Security Trails search engine, the world's largest repository of historical DNS data (Requires an API key, see
  below.) - https://securitytrails.com

• -s, --shodan : Shodan search engine will search for ports and banners from discovered hosts (Requires an API key, see below.)
  https://shodan.io

• Sitedossier : Find available information on a site - http://www.sitedossier.com

• Subdomaincenter : A subdomain finder tool used to find subdomains of a given domain - https://www.subdomain.center/

• subdomainfinderc99 : A subdomain finder is a tool used to find the subdomains of a given domain - https://subdomainfinder.c99.nl

• threatminer : Data mining for threat intelligence - https://www.threatminer.org/

• tomba : Tomba search engine (Requires an API key, see below.) - https://tomba.io

• urlscan : A sandbox for the web that is a URL and website scanner - https://urlscan.io

• venacus : Venacus search engine (Requires an API key, see below.) - https://venacus.com

• vhost : Bing virtual hosts search

• virustotal : Domain search (Requires an API key, see below.) - https://www.virustotal.com

• whoisxml : Subdomain search (Requires an API key, see below.) - https://subdomains.whoisxmlapi.com/api/pricing

• yahoo : Yahoo search engine

• zoomeye : China's version of Shodan (Requires an API key, see below.) - https://www.zoomeye.org

Active modules -

• DNS brute force : dictionary brute force enumeration

• Screenshots : Take screenshots of subdomains that were found

Modules that require an API key -

Documentation to setup API keys can be found at - https://github.com/laramies/theHarvester/wiki/Installation#api-keys

• bevigil - Free upto 50 queries. Pricing can be found here: https://bevigil.com/pricing/osint

• bing

• bufferoverun - uses the free binaAPI

• censys - API keys are required and can be retrieved from your Censys account.

• criminalip

• fullhunt

• github

• hunter - limited to 10 on the free plan, so you will need to do -l 10 switch

• hunterhow

• intelx

• netlas - $

• onyphe -$

• pentestTools - $

• projecDiscovery - invite only for now

• rocketreach - $

• securityTrails

• shodan - $

• tomba - Free up to 50 search.

• venacus - $

• whoisxml

• zoomeye

📕 Step-by-Step Usage Guide -

1. Install the tool

Make sure you have Python 3 installed. Then clone the tool from GitHub by executing following commands one by one :-

Ø git clone https://github.com/laramies/theHarvester.git

Ø cd theHarvester

Ø pip install -r requirements.txt

2. Basic Command Structure

· -d specifies the domain (e.g., example.com)

· -b is the data source (e.g., google, bing, crtsh, anubis, etc.)

3. Sample Usage

Sample output from querying tesla.com with Bing -

· These results give you insight into possible email addresses and subdomains exposed to the public.

Educational Use Cases -

1. Red Team Training: Teaches students how attackers gather intelligence before launching cyberattacks, enhancing threat modeling skills.

2. Reconnaissance Practice in CTFs: Useful in Capture The Flag challenges to identify domains, emails, and services as part of offensive tasks.

3. Information Security Curriculum: Helps demonstrate passive information gathering techniques in university and certification programs.

4. Threat Intelligence Analysis: Introduces the process of gathering open source data to analyze potential threats in a controlled setting.

5. Compliance and Audit Education: Shows how public exposure of sensitive data can lead to audit findings or compliance violations (e.g., GDPR issues).

Limitations

1. Public Data Only: OSINT tools can only collect what is publicly indexed or exposed, private or protected content remains inaccessible.

2. Inconsistent Results Across Sources: Different search engines and APIs may return varying levels of detail depending on rate limits or indexing.

3. Blocked or Rate-Limited Queries: Overuse or malformed queries may trigger captchas, bans, or restrictions from data sources.

4. Lack of Context or Validation: Tools cannot verify accuracy or intent behind discovered data; results may include outdated or unrelated entries.

5. No Post-Exploitation Support: OSINT is limited to information gathering and doesn't include exploitation, lateral movement, or internal scanning.

Ethical Considerations

1. Consent is Critical: Using OSINT on real individuals or organizations without permission, even if data is public, can violate ethical norms.

2. Avoid Targeting Individuals: Gathering personal emails, phone numbers, or social media details without consent raises privacy and legal concerns.

3. Academic Integrity: Misrepresenting OSINT exercises or using real targets without disclosure may result in academic violations.

4. Data Misuse Risks: Information collected via OSINT (e.g., leaked credentials) should never be used maliciously or shared irresponsibly.

Legal Boundaries Vary: Laws differ by country and region (e.g., GDPR in the EU), and using OSINT tools irresponsibly can lead to legal action.

Written by: Keshav Goyal and Deeptansh Nagar

Disclaimer: This post was authored by interns participating in the Infosec Dot Internship Program. Infosec Dot does not verify the accuracy, originality, or authenticity of the content. The views expressed are solely those of the authors and do not necessarily reflect those of Infosec Dot.