Website Information Gathering Tools & Commands

Getting Started

What Is Website Information Gathering?

Website information gathering is the process of collecting information about a target domain, its subdomains, DNS records, IP addresses and web technologies before security testing.

Passive reconnaissance: Collect information from public sources and third-party services without directly probing the target infrastructure.

Active reconnaissance: Send DNS or HTTP requests to discover and verify assets.

Important: Only perform active reconnaissance against infrastructure you own or have permission to assess.

Visual Explanation

How Website Reconnaissance Works

From public domain information to verified web services

Part 01

Passive Information Gathering Tools

Passive tools use public information sources. They help identify possible assets before directly testing the target.

01

WHOIS — Domain Registration Information

What is WHOIS? A lookup method used to find publicly available domain registration information such as registrar, registration dates and nameservers.

Bash · WHOIS

whois example.com

Illustrative Output

Domain Name: EXAMPLE.COM
Registrar: Example Registrar
Name Server: NS1.EXAMPLE.NET
Name Server: NS2.EXAMPLE.NET
Domain Status: active

Example only. Actual WHOIS output depends on the registry and may hide registration details.

ICANN Lookup ↗

02

dig and nslookup — DNS Information

What is DNS lookup? It helps you find IP addresses, mail servers, nameservers and other DNS records.

Bash · DNS Lookup

dig example.com A +short
dig example.com AAAA +short
dig example.com MX +short
dig example.com NS +short
dig example.com TXT +short
nslookup example.com

Example Output — A Record

192.0.2.10

Illustrative IP address from a documentation-only range.

DNS Record	Purpose
A	IPv4 address
AAAA	IPv6 address
MX	Mail servers
NS	Nameservers
TXT	Text records, including SPF information
CNAME	Alias to another domain name

BIND / dig ↗ Google DNS Lookup ↗

DNS lookup can involve direct DNS queries. Querying target-controlled DNS servers is active interaction, even when using dig.

03

Subfinder — Passive Subdomain Enumeration

What is Subfinder? A command-line tool that discovers subdomains from public and third-party data sources.

Install Subfinder

Bash · Installation

go install github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest

Important commands

Bash · Subfinder

subfinder -d example.com
subfinder -d example.com -all -silent
subfinder -d example.com -all -silent -o subfinder.txt

Illustrative Output

www.example.com
api.example.com
mail.example.com
dev.example.com

These are fictional discovery results.

Flag	Meaning
-d	Target domain
-all	Use all configured sources
-silent	Clean output
-o	Save results to a file

Subfinder GitHub ↗

04

RapidDNS — Online Subdomain Search

What is RapidDNS? A website for searching collected DNS records and discovering possible subdomains.

Browser method

Browser URL

https://rapiddns.io/subdomain/example.com

Terminal method

Bash · RapidDNS

curl -s "https://rapiddns.io/subdomain/example.com" | grep -oE '[A-Za-z0-9.-]+[.]example[.]com' | sort -u

Illustrative Output

api.example.com
mail.example.com
www.example.com

Actual results vary. The terminal method depends on RapidDNS's current page structure.

Open RapidDNS ↗

05

crt.sh — Certificate Transparency Logs

What is crt.sh? A public certificate search engine that can reveal subdomains listed in TLS certificates.

Browser method

Browser URL

https://crt.sh/?q=%25.example.com

Terminal method

Bash · crt.sh

curl -s "https://crt.sh/?q=%25.example.com&output=json" | jq -r '.[].name_value' | sort -u

Save results

Bash · Save Results

curl -s "https://crt.sh/?q=%25.example.com&output=json" | jq -r '.[].name_value' | sort -u > crtsh.txt

Illustrative Output

*.example.com
api.example.com
dev.example.com
www.example.com

Certificate entries may contain historical or wildcard names.

Open crt.sh ↗

06

Amass — Additional Passive Enumeration

What is Amass? An attack-surface mapping tool used to discover domains and related assets. Its passive mode uses OSINT data sources.

Installation

Bash · Kali Linux

sudo apt update
sudo apt install amass

Passive enumeration

Bash · Amass

amass enum -passive -d example.com -o amass.txt

Illustrative Output

api.example.com
dev.example.com
mail.example.com

Example only. Amass output and supported flags depend on the installed version.

OWASP Amass GitHub ↗

Part 02

Active Information Gathering Tools

Active reconnaissance sends DNS or HTTP requests to verify and discover assets. Only test systems within your authorized scope.

07

Puredns — DNS Bruteforce and Resolution

What is Puredns? A tool used for DNS bruteforce, resolving discovered subdomains and filtering wildcard DNS responses.

Step 1 — Install Puredns

Puredns requires Go and MassDNS. Install its dependencies first.

Install packages

Bash · Dependencies

sudo apt update
sudo apt install -y git make gcc golang-go

Install MassDNS

Bash · MassDNS

git clone https://github.com/blechschmidt/massdns.git
cd massdns
make
sudo make install

Install Puredns

Bash · Puredns

go install github.com/d3mondev/puredns/v2@latest

Verify installation

Bash · Verify

~/go/bin/puredns --help

If your shell cannot find puredns, add Go's binary directory to PATH:

Bash · Fix PATH

export PATH="$PATH:$(go env GOPATH)/bin"

You also need a suitable DNS resolver list. Download and review one from the resource below, then save it as ~/puredns/resolvers.txt.

Puredns GitHub ↗ MassDNS GitHub ↗ Resolver Lists ↗

Step 2 — Important Puredns Commands

DNS bruteforce

Bash · Bruteforce

puredns bruteforce /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt thepentestlab.in -r ~/puredns/resolvers.txt -l 10 -w resolved.txt

Resolve discovered subdomains

Bash · Resolve

puredns resolve passive.txt -r ~/puredns/resolvers.txt -w resolved.txt

Illustrative Contents of resolved.txt

www.thepentestlab.in
labs.thepentestlab.in
shop.thepentestlab.in

Example format only, not a live scan or verification of these hosts.

Option	Meaning
bruteforce	Test candidate subdomain names
resolve	Resolve existing domain names
-r	DNS resolver list
-l 10	Wildcard-test concurrency limit
-w	Save results

Remember: Check that the wordlist and resolver file exist before running the commands. Use puredns bruteforce --help to verify flags for your version. The -l flag is not a general requests-per-second limit.

08

httpx-toolkit — Find Responding Web Hosts

What is httpx? A command-line HTTP probing tool that checks whether discovered hosts respond over HTTP or HTTPS.

Installation using Go

Bash · Install httpx

go install github.com/projectdiscovery/httpx/cmd/httpx@latest

On Kali Linux, the ProjectDiscovery binary may be named httpx-toolkit. A Go installation commonly uses httpx.

Important command

Bash · HTTP Probing

cat resolved.txt | httpx-toolkit -silent -title -tech-detect -follow-redirects -status-code -rl 15

Save results

Bash · Save Results

cat resolved.txt | httpx-toolkit -silent -title -tech-detect -follow-redirects -status-code -rl 15 -o live-hosts.txt

Illustrative Output

https://www.example.com [200] [Example Home] [Nginx]
https://api.example.com [200] [API] [Node.js]
https://admin.example.com [403] [Forbidden] [Apache]

Fictional output to demonstrate status codes, page titles and technology detection.

Flag	Meaning
-silent	Clean output
-title	Page title
-tech-detect	Detect technologies
-follow-redirects	Follow redirects
-status-code	Show HTTP status code
-rl 15	Maximum 15 requests per second
-o	Save results

httpx GitHub ↗

09

WhatWeb — Website Technology Detection

What is WhatWeb? A tool used to identify possible web technologies, CMS platforms, frameworks and server software.

Installation

Bash · Install WhatWeb

sudo apt update
sudo apt install whatweb

Basic command

Bash · WhatWeb

whatweb https://example.com

Illustrative Output

https://example.com [200 OK] Country[RESERVED] HTML5 Nginx Title[Example Website]

Fictional technology fingerprinting output.

WhatWeb GitHub ↗

Practical Example

Complete Website Information Gathering Workflow

Follow this order when assessing your own domain or another explicitly authorized target.

1. Find Subdomains

Subfinder

subfinder -d thepentestlab.in -silent -o subfinder.txt

2. Combine Passive Results

First save your RapidDNS, crt.sh and Amass results in their corresponding text files.

Combine Results

cat subfinder.txt rapiddns.txt crtsh.txt amass.txt | sort -u > passive.txt

3. Resolve Passive Subdomains

Puredns

puredns resolve passive.txt -r ~/puredns/resolvers.txt -w resolved-passive.txt

4. Run DNS Bruteforce

Puredns Bruteforce

puredns bruteforce /usr/share/seclists/Discovery/DNS/subdomains-top1million-20000.txt thepentestlab.in -r ~/puredns/resolvers.txt -l 10 -w bruteforce.txt

5. Combine Resolved Results

Combine Files

cat resolved-passive.txt bruteforce.txt | sort -u > resolved.txt

6. Check Responding Web Hosts

httpx-toolkit

cat resolved.txt | httpx-toolkit -silent -title -tech-detect -follow-redirects -status-code -rl 15 -o live-hosts.txt

Note: This workflow assumes the required tools, resolver list and input files already exist. Keep the discovered assets within the authorized testing scope.

Quick Revision

Website Reconnaissance Tools Summary

Order	Tool	Main Purpose
1	WHOIS	Domain registration information
2	dig / nslookup	DNS records
3	Subfinder	Passive subdomain discovery
4	RapidDNS	Public DNS information
5	crt.sh	Certificate records
6	Amass	Additional passive enumeration
7	Puredns	DNS bruteforce and resolution
8	httpx	Responding web hosts
9	WhatWeb	Technology fingerprinting

Key takeaway: Start with public domain and DNS information, discover possible subdomains, verify DNS resolution and identify responding web services.

Website Information Gathering: Passive & Active Reconnaissance Tools

What Is Website Information Gathering?

How Website Reconnaissance Works

Passive Information Gathering Tools

WHOIS — Domain Registration Information

dig and nslookup — DNS Information

Subfinder — Passive Subdomain Enumeration

RapidDNS — Online Subdomain Search

crt.sh — Certificate Transparency Logs

Amass — Additional Passive Enumeration

Active Information Gathering Tools

Puredns — DNS Bruteforce and Resolution

Step 1 — Install Puredns

Step 2 — Important Puredns Commands

httpx-toolkit — Find Responding Web Hosts

WhatWeb — Website Technology Detection

Complete Website Information Gathering Workflow

1. Find Subdomains

2. Combine Passive Results

3. Resolve Passive Subdomains

4. Run DNS Bruteforce

5. Combine Resolved Results

6. Check Responding Web Hosts

Website Reconnaissance Tools Summary

Next Lesson: Content Discovery