Reducing Your Own Public Footprint#

Open Source Intelligence (OSINT) is the practice of finding and connecting publicly available information, such as DNS records, TLS certificate logs, search results, code repositories and document metadata, to answer a specific question. Defenders use it most valuably as a mirror: run the same public lookups an outside party could run against your own organisation, find what you are unintentionally exposing, and fix it before it becomes a problem. Threat-intelligence teams use the same sources to understand adversaries, and authorised penetration testers use them for reconnaissance within an agreed scope.

This lesson treats OSINT as a defensive, self-assessment discipline: what is publicly discoverable about your own estate, and how to shrink it.

Passive vs Active#

Passive Active
What it queries Third-party sources: certificate logs, search engines, public archives Your target's own systems directly
Appears in the target's logs No Yes
Authorisation Legitimate purpose and privacy rules still apply Always required, in writing
Examples crt.sh, WHOIS, public scan-engine lookups port scanning, crawling, probing endpoints

OSINT is mostly passive: it reads what third parties already publish. The moment you send traffic to a target's own infrastructure you are doing active testing, which needs explicit authorisation, as covered in penetration testing. For your own self-audit, active steps against your own systems are fine with your organisation's approval; against anyone else they are not.

The Main Public Sources#

  • Certificate Transparency (CT) logs. Every publicly trusted TLS certificate is logged, so certificate records reveal hostnames, including internal-sounding ones a team forgot were public.
  • DNS. Public records, and mail records (MX, SPF, DMARC) that show which email and SaaS providers you use. See DNS.
  • WHOIS / registration data. Registrar and dates; many contact fields are now redacted for privacy.
  • Internet-wide scan engines (such as Shodan and Censys) index services that respond on the public internet, so an exposed panel or an outdated appliance shows up in their results.
  • Public code hosting. Repositories sometimes contain committed keys, internal hostnames or configuration that reveal architecture.
  • Document metadata. Published PDFs and images can carry author names, software versions and internal file paths in their metadata.
  • Search engine operators. Queries like site:example.com filetype:pdf surface documents an organisation did not mean to leave indexed.

A Worked Example: Enumerate Your Own Subdomains From CT Logs#

Certificate Transparency is the highest-value defensive source because it shows every name you have ever put a public certificate on, which is often more than any inventory spreadsheet lists. The crt.sh service exposes this as JSON:

import requests

def subdomains_from_ct(domain: str) -> set:
    # crt.sh returns one record per matching certificate; % is the SQL wildcard
    r = requests.get("https://crt.sh/", params={"q": f"%.{domain}", "output": "json"}, timeout=30)
    r.raise_for_status()
    names = set()
    for row in r.json():
        for name in row["name_value"].splitlines():   # a cert can list several names
            name = name.strip().lstrip("*.").lower()
            if name.endswith(domain):
                names.add(name)
    return names

for host in sorted(subdomains_from_ct("example.com")):
    print(host)

Run this against a domain you own. The output is the list of every subdomain that has appeared on a certificate: production sites, but also the staging, vpn-old, jenkins and admin names that are the interesting ones. Each result is a question for your own review: should this still exist, is it patched, should it be reachable from the public internet at all? For each live host you then check whether it resolves, what it serves, and whether it belongs on the internet, and you retire or firewall the ones that do not.

The next defensive step is to see which of those names still resolve, using only public DNS (this reads DNS, it does not touch the hosts themselves):

import socket

def resolves(host: str):
    try:
        return socket.gethostbyname(host)
    except socket.gaierror:
        return None

for host in sorted(subdomains_from_ct("example.com")):
    ip = resolves(host)
    print(f"{host:40} {ip or '(no DNS record)'}")

Names with no DNS record are stale certificate history (still worth noting, as a name that resolves again later could be a dangling-DNS takeover risk). Names that resolve are your live external footprint to review.

Turning Findings Into Fixes#

The defensive value of OSINT is the remediation list it produces. A typical self-audit yields:

  • Forgotten hosts: an old staging or admin subdomain, still live and unpatched. Take it offline or firewall it.
  • Exposed management interfaces: a database console or CI server reachable from the internet. Move it behind a VPN or identity-aware proxy (zero trust).
  • Leaked configuration in public repositories: rotate any exposed credential immediately (see secrets management) and remove the file.
  • Over-informative metadata: strip author names, internal paths and software versions from published documents.
  • Look-alike domains: register-and-monitor for domains that impersonate your brand, which are used to phish your staff and customers.

Doing this on a regular cadence, for example half a day each quarter re-running the same queries against your own estate, is one of the cheapest attacker-simulation exercises an organisation can run, and its outputs (the exposed-host list, the leaked-credential findings, the impersonating-domain report) are exactly what an incident response team reaches for when a real investigation begins.

OSINT is information you are permitted to see: public records, indexed pages, data people published themselves. It becomes unlawful the moment you cross into what is not public or not authorised, such as accessing anything behind a login you were not given, or bypassing a protection to reach a restricted resource. Two boundaries matter in practice:

  • Authorisation. Reading third-party public sources about your own organisation is fine. Any active step against systems you do not own requires written permission, the "rules of engagement" from the penetration testing lesson. "It was public" is not a legal defence for unauthorised access.
  • Personal data. Information about identifiable people is regulated (GDPR and similar laws) even when it is publicly visible. Collecting and storing it needs a lawful basis and a real purpose; "it was on the internet" does not exempt you.

Keep every engagement scoped, authorised and documented, and default to your own estate when practising.

Practitioner Discipline#

  • Document each step. Your chain of evidence is the deliverable; record the query, the source and the timestamp.
  • Preserve raw results before interpreting them (the original JSON or screenshot, not just your summary), so a conclusion can be re-checked.
  • Re-verify at the source anything you plan to act on. A single stale record can send an investigation down the wrong path.
  • Prefer authoritative sources over aggregators when accuracy matters.

Practice#

Run the CT-log script against a domain your organisation owns (or your own personal domain). Produce three lists: hosts that resolve, hosts that no longer resolve, and hosts you did not know existed. For each live host, note whether it should be public and whether it is patched, and turn that into a short remediation list. Then check whether any published document on your site leaks metadata by inspecting its properties. Do this only against assets you own or are authorised to assess.