Photon: Incredibly fast crawler designed for OSINT.

🧭 Overview

Photon is a fast OSINT-focused web crawler written in Python by s0md3v. Point it at a domain and, in a single crawl, it harvests a remarkable amount of intel: in-scope and out-of-scope URLs, parameterized (fuzzable) URLs, emails, social accounts, AWS buckets, files (PDF/XML/…), JavaScript files, secret keys, subdomains, and DNS data.

Unlike a pure link crawler, Photon is built to extract intelligence while it walks the site. It organizes findings into neat text files on disk (internal.txt, fuzzable.txt, keys.txt, …), can export to JSON/CSV, and even clones sites locally. For a tester, that means one command produces both an attack-surface map and a ready-to-fuzz parameter list.

| 🔐 OffSec Tip
Photon’s superpower is that it sorts what it finds. Instead of one giant URL dump, you get fuzzable.txt (params), scripts.txt (JS), keys.txt (secrets), and intel.txt (emails/social) — already split for you.

Repo: github.com/s0md3v/Photon

📦 Installation

Photon is a Python 3 script. The canonical install is a git clone:

1
2
3
4

git clone https://github.com/s0md3v/Photon.git
cd Photon
pip install -r requirements.txt
python photon.py --help

Docker:

1
2
3
4
5
6

git clone https://github.com/s0md3v/Photon.git
cd Photon
docker build -t photon .

# Run (mount a volume so output survives the container)
docker run -it --name photon -v "$PWD:/Photon/google.com" photon:latest -u google.com

| 🔐 OffSec Tip
Run it from inside the cloned folder as python photon.py. Depending on your setup you may need python3 photon.py.

🚀 Basic Usage

The only required flag is -u (the target URL):

1

python photon.py -u "https://example.com"

Photon crawls the target and drops a folder named after the domain (e.g. example.com/) containing all the result files.

🚩 All CLI Flags

🕷️ Core crawl

🔐 Authentication & requests

🌱 Scope & seeds

⛏️ Data extraction

📤 Output

🧰 Misc

📖 Every Flag Explained (the ones that matter)

📏 -l, --level — crawl depth

Depth 2 (default) crawls the homepage and its seeds, then the results of those. Bump it for deeper coverage; each level multiplies the work.

1

python photon.py -u "https://example.com" -l 3

⚡ -t, --threads + -d, --delay — speed vs. stealth

More threads = faster, but noisier and more likely to trip rate limits. Add a delay to slow down.

1

python photon.py -u "https://example.com" -t 10 -d 1

🔑 --keys — the money flag

Turns on secret/high-entropy extraction, writing hits to keys.txt. Cheap to add, frequently pays off.

1

python photon.py -u "https://example.com" --keys

🧩 -r, --regex — grab anything you want

Extract custom patterns while crawling — internal IDs, phone numbers, API paths, whatever:

1

python photon.py -u "https://example.com" --regex "\d{10}"

🕰️ --wayback — free historical seeds

Seeds the crawl with archive.org URLs for the current year, so you also touch old/forgotten paths:

1

python photon.py -u "https://example.com" --wayback

🚫 --exclude — stay in scope

Regex out logout links, media, or out-of-scope sections:

1

python photon.py -u "https://example.com" --exclude "/(logout|static|assets)/"

🌐 --dns — subdomains + graph

Enumerates subdomains and DNS data, saving subdomains.txt and a DNS visualization:

1

python photon.py -u "https://example.com" --dns

🔗 --only-urls — fast link map

Skip all intel/JS extraction when you just want the URL surface quickly:

1

python photon.py -u "https://example.com" --only-urls

🗂️ Output Files Explained

Photon writes a folder (named after the domain, or your -o) containing:

| 🔐 OffSec Tip
fuzzable.txt is the file you want most often — it’s a pre-filtered list of parameterized URLs, ready to hand to ffuf, qsreplace, or your injection tooling.

🧪 Practical Examples

Authenticated crawl:

1

python photon.py -u "https://example.com" -c "PHPSESSID=u5423d78fqbaju9a0qke25ca87"

Add extra starting points:

1

python photon.py -u "https://example.com" -s "https://example.com/blog/2018,https://example.com/portal"

Custom user-agents:

1

python photon.py -u "https://example.com" --user-agent "curl/7.35.0,Wget/1.15"

Deeper, threaded crawl with a timeout tweak:

1

python photon.py -u "https://example.com" -l 3 -t 10 --timeout 4

🧠 Advanced Examples

🛰️ Full intel sweep

Depth 3, keys on, wayback seeds, DNS enumeration, verbose:

1

python photon.py -u "https://example.com" -l 3 --keys --wayback --dns -v

📥 Export for tooling

Dump results as JSON so you can jq them:

1

python photon.py -u "https://example.com" --keys --export json

🧬 Clone for offline analysis

Save every crawled page locally (useful for source review without re-hitting the target):

1

python photon.py -u "https://example.com" --clone

➡️ Pipe a single result set out live

--stdout prints exactly one variable to stdout so you can chain it. Supported values:
files, intel, robots, custom, failed, internal, scripts, external, fuzzable, endpoints, keys.

1

python photon.py -u "https://example.com" --stdout fuzzable | qsreplace FUZZ

🎯 Real Pentest Workflow

From a single domain to a fuzzed, JS-mined attack surface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# 1. Deep crawl with intel + secrets + wayback seeds
python photon.py -u "https://target.com" -l 3 -t 10 --keys --wayback -o photon_out

# 2. Confirm which discovered URLs are actually live
cat photon_out/internal.txt | httpx -silent -mc 200,301,302,403 > live.txt

# 3. Fuzz the parameterized URLs Photon already isolated for you
cat photon_out/fuzzable.txt | qsreplace "FUZZ" \
  | while read u; do ffuf -w payloads.txt -u "$u" -mc 200,500; done

# 4. Deep-dive the JavaScript Photon collected
cat photon_out/scripts.txt | httpx -silent | while read u; do jsluice urls "$u"; done

# 5. Review the intel + keys by hand
cat photon_out/keys.txt photon_out/intel.txt

| 🔐 OffSec Tip
Photon and jsluice pair beautifully: Photon finds the JS files (scripts.txt), jsluice dissects them (endpoints, secrets, params). Run them back to back.

🤖 Automation Examples

Reusable function:

1
2
3
4
5
6

ph() {
  # usage: ph target.com
  python /opt/Photon/photon.py -u "https://$1" -l 3 --keys --wayback -o "photon_$1"
  echo "[*] fuzzable URLs:"
  wc -l "photon_$1/fuzzable.txt"
}

Loop over a scope file:

1
2
3
4

while read domain; do
  echo "[*] Crawling $domain"
  python photon.py -u "https://$domain" --keys -o "out/$domain"
done < scope.txt

🧾 JSON / Output Explanation

Photon’s default output is a set of plain-text files (see the table above) — not JSON. That’s deliberate: each file is already a clean, deduplicated, single-purpose wordlist you can pipe directly (cat fuzzable.txt | ...).

When you do want structured data, add --export json (or --export csv) and Photon writes a formatted export you can parse with jq. Use --stdout <variable> when you want a single category streamed live into another tool without touching disk.

🔧 jq Filtering Examples

After --export json:

1
2
3
4
5
6
7
8

# Pretty-print the export
jq '.' photon_out/exported.json

# Pull just the fuzzable URLs (adjust key to match your export)
jq -r '.fuzzable[]?' photon_out/exported.json

# Merge internal + external into one deduped list
jq -r '.internal[]?, .external[]?' photon_out/exported.json | sort -u

| 🔐 OffSec Tip
Export key names depend on the Photon version. Run jq 'keys' on the export first to see exactly what’s available before scripting.

📡 Combine with httpx

Photon finds URLs; httpx tells you which ones live and what they return:

1

cat photon_out/internal.txt | httpx -silent -sc -title -mc 200,301,302,401,403

🕰️ Combine with gau

Feed Photon’s crawl straight into historical URL mining for extra coverage (Photon also has --wayback built in, but gau casts a wider net):

1

cat photon_out/internal.txt | unfurl domains | sort -u | gau --subs

📜 Combine with subjs

Widen JavaScript discovery beyond what Photon’s crawl reached:

1

cat photon_out/internal.txt | subjs | sort -u >> photon_out/scripts.txt

🧃 Combine with jsluice (the natural pair)

Let Photon collect scripts, let jsluice extract from them:

1
2
3

cat photon_out/scripts.txt \
  | httpx -silent -mc 200 \
  | while read u; do jsluice urls -R "$u" "$u"; jsluice secrets "$u"; done

💥 Combine with ffuf

fuzzable.txt is basically a ready-made target list:

1
2
3
4

cat photon_out/fuzzable.txt | qsreplace FUZZ | sort -u \
  | while read target; do
      ffuf -w payloads.txt -u "$target" -mc 200,301,302,403,500
    done

⚡ One-liners

1
2
3
4
5
6
7
8

# Crawl + immediately probe which internal URLs are live
python photon.py -u https://target.com --only-urls --stdout internal | httpx -silent -mc 200

# Crawl, then fuzz every parameterized URL Photon isolated
python photon.py -u https://target.com && cat target.com/fuzzable.txt | qsreplace FUZZ | httpx -silent

# Photon -> jsluice in one breath
python photon.py -u https://target.com --stdout scripts | httpx -silent | while read u; do jsluice urls "$u"; done

⚠️ Common Mistakes

• Forgetting --keys. Secret extraction is off by default. If you skip it, keys.txt never gets written.
• Cranking -t too high. Photon defaults to 2 threads for a reason — pushing to 50 on a fragile target gets you blocked or drops results into failed.txt.
• Expecting JSON by default. Base output is .txt files. Add --export json if you need structured data.
• Ignoring fuzzable.txt. People re-derive a parameter list with other tools when Photon already handed them one.
• No --exclude on big sites. Without scope control the crawler wanders into logout links and infinite calendars.
• Wrong Python. Photon is Python 3; running it under a stale Python 2 environment will error out.

💡 Pro Tips & Performance

• Start shallow, then go deep. Run -l 2 --only-urls first for a quick map, then a full -l 3 --keys --dns pass on the interesting hosts.
• --wayback for free coverage. It costs almost nothing and often surfaces retired endpoints the live site no longer links to.
• Tune threads to the target. -t 10 on a robust CDN-backed site; keep it low (-t 2 -d 1) on something fragile or WAF’d.
• Use --stdout to avoid disk churn in pipelines — stream one category straight into the next tool.
• Chain into jsluice every time. scripts.txt → jsluice is the highest-value follow-up Photon enables.
• Keep the output folder per target with -o so parallel scans don’t clobber each other’s files.

🏁 Conclusion

Photon is the OSINT crawler that does your sorting for you. One command turns a domain into a labeled set of files — in-scope URLs, fuzzable parameters, JavaScript, secrets, subdomains, and intel — each already deduplicated and ready to pipe. That structure is what makes it such a good first recon step: everything downstream (httpx for liveness, ffuf for fuzzing fuzzable.txt, jsluice for dissecting scripts.txt) has a clean input waiting for it.

Remember the flags that matter — --keys for secrets, --wayback for historical reach, --dns for subdomains, -l/-t/-d to balance depth against stealth — and let the output files drive the rest of your workflow. Pair it with jsluice and you’ve got a JavaScript-to-endpoints pipeline that’s hard to beat.

| 🔐 OffSec Tip
The fastest way to level up your recon: run Photon first, then feed its scripts.txt to jsluice and its fuzzable.txt to ffuf. Three tools, one clean chain.