Content Discovery in Web Pentesting: Tools, Commands & Examples

What Is Content Discovery?

Content discovery is the process of finding website pages, directories, files, URLs and API endpoints that may not be visible in the main navigation.

For example, a website may have a public homepage, but also contain a login page, API routes and old files.

https://example.com/login
https://example.com/admin
https://example.com/api/users
https://example.com/backup.zip

These are illustrative URLs, not confirmed resources on example.com.

How Content Discovery Works

Manual Content Discovery

Start by checking website navigation, page source, links, robots.txt and sitemap.xml.

robots.txt

This file gives instructions to web crawlers. It may mention paths that are not linked from the homepage.

curl -s https://example.com/robots.txt

User-agent: *
Disallow: /private/
Disallow: /admin/

sitemap.xml

A sitemap lists URLs that a website wants search engines to discover.

curl -s https://example.com/sitemap.xml

<urlset>
  <url>
    <loc>https://example.com/about</loc>
  </url>
  <url>
    <loc>https://example.com/contact</loc>
  </url>
</urlset>

Wayback Machine — Historical URL Discovery

What is Wayback Machine? An Internet Archive service that stores historical snapshots of websites.

It can help you discover old pages, endpoints and URL structures.

Browser Method

https://web.archive.org/web/*/example.com/*

Terminal Method — CDX API

curl -sG "https://web.archive.org/cdx/search/cdx" --data-urlencode "url=example.com/*" --data-urlencode "output=txt" --data-urlencode "fl=original" --data-urlencode "collapse=urlkey" | sort -u > wayback.txt

https://example.com/login
https://example.com/old-dashboard
https://example.com/api/v1/users

Archived URLs may be unavailable today. API access may also be rate-limited or unavailable.

gau — Get All URLs

What is gau? A tool that collects known URLs from public sources such as Common Crawl, Wayback Machine and other configured providers.

Installation

go install github.com/lc/gau/v2/cmd/gau@latest

If Go's binary directory is not in PATH, run the installed binary from ~/go/bin/gau.

Important Commands

Find known URLs

gau example.com

Save results

gau example.com --o gau.txt

Exclude common static file extensions

gau example.com --blacklist png,jpg,jpeg,gif,svg,woff,woff2

https://example.com/login
https://example.com/search?q=test
https://example.com/api/products
https://example.com/account/settings

Katana — Website Crawling

What is Katana? A web crawler that visits pages, follows links and discovers URLs and endpoints.

Installation

go install github.com/projectdiscovery/katana/cmd/katana@latest

Important Commands

Basic crawling

katana -u https://example.com

Crawl with depth and rate limit

katana -u https://example.com -d 2 -rl 5 -silent -o katana.txt

Use a list of websites

katana -list live-urls.txt -d 2 -rl 5 -silent -o crawled.txt

https://example.com/
https://example.com/login
https://example.com/products
https://example.com/api/products
https://example.com/assets/app.js

Gobuster DIR — Directory and File Discovery

What is Gobuster DIR? A tool that uses a wordlist to test possible directory and file paths on a website.

Installation

sudo apt update
sudo apt install gobuster

Important Commands

Basic directory discovery

gobuster dir -u https://example.com -w /usr/share/wordlists/dirb/common.txt -t 5

Discover files with extensions

gobuster dir -u https://example.com -w /usr/share/wordlists/dirb/common.txt -x php,txt,html -t 5 -o gobuster.txt

/admin     (Status: 403)
/login     (Status: 200)
/images    (Status: 301)
/robots.txt (Status: 200)

ffuf — Web Content Fuzzing

What is ffuf? A web fuzzing tool that replaces the FUZZ keyword with values from a wordlist.

It is useful for discovering paths, files and, when authorized, testing parameter names or values.

Installation

sudo apt update
sudo apt install ffuf

Important Commands

Directory discovery

ffuf -u https://example.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -rate 5

Filter 404 responses

ffuf -u https://example.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404 -rate 5

Save results

ffuf -u https://example.com/FUZZ -w /usr/share/wordlists/dirb/common.txt -fc 404 -rate 5 -o ffuf.json -of json

admin  [Status: 403, Size: 275]
login  [Status: 200, Size: 1830]
images [Status: 301, Size: 310]

Content Discovery Workflow

Use this sequence on a website you own or have permission to test.

1. Collect Historical URLs

gau thepentestlab.in --o gau.txt

2. Crawl the Website

katana -u https://thepentestlab.in -d 2 -rl 5 -silent -o katana.txt

3. Combine Discovered URLs

cat gau.txt katana.txt | sort -u > discovered-urls.txt

4. Discover Additional Paths

gobuster dir -u https://thepentestlab.in -w /usr/share/wordlists/dirb/common.txt -t 5 -o gobuster.txt

5. Fuzz Paths with ffuf

ffuf -u https://thepentestlab.in/FUZZ -w /usr/share/wordlists/dirb/common.txt -rate 5 -fc 404 -o ffuf.json -of json

6. Review Results

Compare discovered URLs with the website's current pages. Check which resources exist, their response codes and whether they belong to your authorized scope.

Content Discovery Tools Summary

Key takeaway: Historical URL collection helps find previously known content. Crawling follows existing links, while wordlist-based discovery tests paths that may not be linked anywhere.