SCIENTIFIC SPECIFICATION
Observatory Methodology & Privacy Framework
A transparent, reproducible, and non-monetized framework for measuring machine navigation on the web.
1. Why User-Agent Strings Alone Cannot Be Trusted
In HTTP networking, the User-Agent header is declared entirely by the client. Any developer or script can send User-Agent: GPTBot/1.2 without any connection to OpenAI.
The Ghost Traffic Observatory enforces a strict multi-tier evidence hierarchy:
| Classification Level | Evidence Requirement | Interpretation |
|---|---|---|
| Verified Vendor Bot | Declared commercial User-Agent PLUS matching documented vendor IP CIDR or reverse DNS verification. | Confirmed official crawler operated by OpenAI, Anthropic, Google, Microsoft, Meta, etc. |
| Claimed Bot | Recognized bot User-Agent string, but originating from an unverified IP range. | Possibly spoofed or third-party proxy using an official bot signature. |
| Goal-Directed Automation | Repeated correct navigation choices through the deterministic or semantic benchmark. | Autonomous agent actively solving decision logic. |
| Suspected Crawler | Traversal characteristics (Python/Scrapy UA, automated rapid requests, absence of JS beacon). | Automated script or scraper without declared identity. |
| Likely Human | Browser headers (Sec-CH-UA, Accept-Language), executed JS beacon, asset waterfall. |
Interactive human user. |
2. Zero-PII Transient IP Verification Architecture
Privacy is guaranteed at the edge architecture level. The observatory processes IP addresses strictly transiently:
- When an HTTP request arrives, the edge function performs an in-memory bitwise comparison of the IP against vendor CIDR blocks.
- The boolean result (
isVerified: true/false) is recorded. - The raw IP address is immediately discarded from memory. It is never logged to disk, never written to SQLite/D1, and never exported in datasets.