What it requests, and what it refuses to
The scanner requests three things, all of them already public: your /robots.txt, your /products.json storefront feed, and a handful of product pages. It also probes /llms.txt and /agents.md, the two files an AI agent looks for to find out where your catalog lives.
It identifies itself as SKUcoverage/0.1, follows at most two redirects, re-checks every redirect target against the same host rules, and stops reading a response at a fixed byte cap.
It never logs in, never touches admin or checkout, never submits a form, and never reads a Storefront API token out of your theme source. It changes nothing.
One store can be scanned once every 45 seconds across all visitors, so this page cannot be pointed at a competitor as a load generator.
How it honours robots.txt
The scanner implements RFC 9309 rather than using a naive parser, because the two properties that matter are exactly the ones naive parsers get wrong.
- Longest match wins, not first match. Shopify's own robots.txt opens the * group with Allow: /, which under first-match-wins would shadow every Disallow after it. A first-match parser will cheerfully tell you /admin is crawlable.
- Percent-encoded wildcards are wildcards. Shopify writes Disallow: /cdn/wpm/%2A.js. Read literally that is three characters; read as intended it is a wildcard. Where a strict spec reading and obvious intent disagree, we take whichever is more conservative about someone else's server.
- Groups are merged, not shadowed. §2.2.1 requires that when more than one group matches a crawler, the matching groups are combined. This is not a corner case: Cloudflare's managed robots.txt prepends its own User-agent: * group, so on those zones a merchant's own group is the second one. A parser that keeps only the first silently discards the merchant's entire file and reports every path as crawlable.
If your file disallows the feed, the scan says so and stops, instead of fetching it anyway. The same gate binds the llms.txt and agents.md probes.
Why the free scan shows no grade
The engine scores five weighted pillars out of 100:
- 35
- Attribute coverage — the typed product details agents filter on
- 20
- Category resolvability against the Shopify Standard Product Taxonomy
- 20
- Product identifiers — GTIN, SKU, brand
- 15
- Structured data — Product JSON-LD and its offer fields
- 10
- Listing content — description depth, named options
The free scan runs on a Cloudflare Worker with a 10 ms CPU budget per request, and raising it is a paid feature. The two heaviest pillars both need the full Shopify taxonomy — a 2 MB index that cannot be loaded inside that budget. So the free scan measures identifiers, structured data and listing content: 45 of 100 points.
45 is below the engine's evidence floor of 60, so this endpoint returns counts of concrete facts and never a score or a letter grade. The constant that enforces it is PORTABLE_CAN_GRADE, and a test fails the build if it ever silently flips. Headlining a grade built on partial evidence is precisely the failure mode the audit exists to find in other people's catalogs.
How the scan is split across requests
Because each HTTP request gets its own 10 ms budget, the scan is deliberately fragmented rather than run as one job:
- /api/scan/start — reads robots.txt, probes the two agent files, and mints a signed ticket that pins one hostname.
- /api/scan/feed — one page of 50 products per request, up to three pages.
- /api/scan/pdp — one sampled product page per request, up to five, spread across the catalog rather than taken from the front.
The ticket is HMAC-signed rather than stored, so no session state is needed, and it pins the host so a solved challenge cannot be replayed against a different store. Between page fetches the scanner waits out your Crawl-delay if you publish one.
The three rules the engine cannot break
Every one of these exists because the opposite behaviour shipped once and produced a confident wrong answer.
- Never award points for a check that did not run. A skipped check is recorded as skipped, with its reason, and is excluded from the denominator — not scored as a pass.
- Never assert a failure from evidence never gathered. Scoring identifiers against a product page that was never fetched would report "no variant carries a SKU" from nothing. So the pillar is either scored against real feed facts or reported as not assessed. A GTIN is only ever judged from a product page's structured data, never from the feed's barcode field, which is why GTIN counts on the results are denominated in pages sampled rather than products listed.
- Never headline a grade built on thin evidence. See the section above.
Why identifiers and structured data decide inclusion
An agent building a candidate set does not read your product description first. It filters on machine-readable fields: a GTIN to match your item to the same item elsewhere, a brand to satisfy a brand query, a category to enter the right comparison at all, and typed attributes to survive a request like "waterproof, size 11, under $150".
A product page that reads beautifully to a human and carries no gtin13, no itemCondition and a Title / Default Title placeholder option is invisible to that filter. That is what "coverage" means here, and why the metric is a count of facts rather than an opinion about your copy.
What we store
Per store: the hostname, the time of the last scan, a scan count, and aggregate non-identifying counts used for the published statistics. Per lead: the email address and the domain scanned. No names, no IP addresses, no product data retained after the response is rendered.
The published percentages are computed server-side from the scanner's own measurements. No request body can write to that table — an endpoint that accepted client-supplied statistics would let anyone poison a trust claim. Percentages are withheld entirely until enough stores have been measured to make one meaningful.