Published 2026-09-22 — v1.0

AI crawler policy + AI pixel methodology

CipherCue observes two public-by-construction facts about how a site treats AI systems: which named AI crawlers it allows or disallows in its own robots.txt, and whether the markup it serves ships a named AI vendor's measurement pixel. This page specifies both signals, the rule dictionary, the recorded fact shape, and false-positive policy.

A crawler-policy observation records the directive a site publishes in its own robots.txt, read as the robots.txt convention intends. A pixel observation records that a named vendor's loader tag is present in the served markup. Neither is an assertion about how a site uses AI, whether the pixel fires for any given visitor, or the site's compliance with any regulation.

Signal 1 — AI crawler policy (robots.txt)

CipherCue fetches https://<domain>/robots.txt and parses its user-agent groups per the robots.txt grouping convention. For each named AI crawler in the dictionary, the most specific matching group is evaluated: a group naming that exact user-agent token overrides the User-agent: * wildcard group. A crawler is recorded as blocked when the applicable group disallows the site root (Disallow: /), and allowed otherwise.

Two boundary cases are recorded explicitly:

No robots.txt — a non-200 response (commonly 404) means the site publishes no directives, so every crawler is implicitly allowed. This is recorded as robots_status = "no_robots".
Named override — a site can allow the wildcard but name a specific AI crawler to block it, or vice versa. The recorded matched_explicitly flag distinguishes a directive that names the crawler from one inherited from the wildcard group.

Signal 2 — AI measurement pixel (served markup)

CipherCue fetches the site root over HTTP and matches its served HTML and inline scripts against named AI vendor pixel loaders. A match observes that the loader tag is present in the markup the site returns. Where the loader carries a public pixel or project identifier in its initialisation call, that identifier is recorded.

This is a static observation of served markup. CipherCue does not execute third-party JavaScript in a browser, so a loader that is injected only at runtime by an unrelated cross-origin script is out of scope. The recorded fact is "this loader tag was present in the served markup on this date", nothing more.

Rule dictionary

The current dictionary lives in config/ciphercue/ai_crawler_rules.php. Crawler rules have the shape:

[
    'token' => 'GPTBot',
    'vendor' => 'OpenAI',
    'purpose' => 'training',   // training | search | user_action
]

Pixel rules have the shape:

[
    'id' => 'openai-measurement-pixel',
    'vendor' => 'OpenAI',
    'product' => 'Measurement Pixel',
    'pattern' => '#bzrcdn\.openai\.com/sdk/oaiq\.min\.js#i',
    'id_regex' => '/.../',    // optional, extracts a public pixel id
]

Adding a crawler token or pixel rule is a documentation change and a methodology version bump.

Recorded facts

Crawler policy — one ai_crawler_policy fact per domain per scan day, holding the full per-crawler observation list:

host — the domain observed
robots_status — reachable or no_robots
observations — per crawler: token, vendor, purpose, status (allowed/blocked), matched_explicitly
blocked_count / total_known_bots
authority_source = "CipherCue AI-crawler rule set", authority_url = this page

Pixel presence — one ai_pixel_observation fact per matched pixel per scan day:

host, final_url
vendor, product, rule_id
pixel_id — the public identifier, where present in the loader
authority_source, authority_url = this page

Pre-disclosure handling

Both signals are derived from public-by-construction resources — a published robots.txt and served page markup — and are not subject to the 48-hour silent disclosure window. They are written with disclosure_status = "public_observation" and surfaced immediately.

False-positive policy

Crawler tokens — matching is on the exact user-agent token, case-insensitive. A site that blocks a search-purpose token (e.g. OAI-SearchBot) but allows a training-purpose token (e.g. GPTBot) is recorded per token; the two are not conflated.
Path-scoped disallows — a Disallow that targets a subpath (e.g. /admin) but not the root is recorded as allowed, because the crawler may still fetch the site's content. Only a root disallow is recorded as blocked.
Pixel loaders — patterns are anchored to vendor-owned loader hosts and file names. A match is the loader tag in served markup, not proof the pixel is active for any visitor.

Correction

If an observation is wrong, email corrections@ciphercue.com. We investigate within 7 days. Adding a crawler token or pixel rule to the dictionary is welcomed as a community contribution.

Changelog
v1.0 — 2026-09-22 — Initial publication. 19 AI crawler tokens across training, search, and user-action purposes; 1 AI pixel rule.